Inference Gateway
docs.inference-gateway.com
- Free plan
- Windows
- macOS
- Linux
- Web
- API
- Self-hosted
What Inference Gateway is
Inference Gateway is an open-source, cloud-native proxy that gives applications one OpenAI-compatible API for hosted and local LLM providers. Its provider list includes OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Google, Mistral, NVIDIA, and others. Applications can change providers through configuration without a redeploy, while model aliases can route requests to pools of upstream deployments. The gateway can discover tools on connected MCP servers and execute calls through the gateway. Agent-to-Agent support enables agents to discover capabilities, delegate tasks, and stream results; agents can be defined in YAML, with a CLI that generates Go, Rust, or TypeScript projects. A Kubernetes Operator offers custom resources for gateways, agents, MCP servers, and chat orchestrators. Documentation covers OIDC authentication, Prometheus-compatible metrics, OpenTelemetry, and structured JSON logs, with Grafana, Loki, and Jaeger examples. Docker and Kubernetes deployment are documented. The project says it can be self-hosted, including in air-gapped environments, and has no analytics or telemetry phoning home. It is released under Apache-2.0.
Who it suits
It suits teams that want one API for multiple hosted and local LLM providers, with configuration-based switching and model routing. Operators who need self-hosting, MCP connections, or Kubernetes management can also consider it.
Good to know, and what to watch
- One OpenAI-compatible API covers hosted and local providers.
- Provider changes do not require an application redeploy.
- Supports MCP tools and agent coordination.
- Documents Docker and Kubernetes deployment.
- Released under Apache-2.0.
- Desktop releases are not signed with Apple or Windows certificates.
The verdict
Inference Gateway offers a broad integration surface for teams that want to route among multiple LLM providers and operate the gateway themselves. Review its deployment and signing details against your environment before adopting it.
What the maker publishes
- Agent coordination
- Its Agent-to-Agent support lets agents discover capabilities, delegate tasks, and stream results within a conversation. source
- Agent workflows
- The product supports Agent-to-Agent coordination and describes defining agents in YAML with a CLI that generates Go, Rust, or TypeScript projects. source
- Authentication
- OIDC authentication validates JWTs from a configured identity provider, and the gateway documentation says the gateway needs no client secret for token validation. source
- Authentication
- Authentication supports OIDC identity providers, with JWT validation against the issuer’s JWKS. source
- Deployment
- The documentation provides a Docker image and describes running the gateway in Docker or Kubernetes. source
- Deployment
- The getting-started guide documents Docker, Docker Compose examples, and Kubernetes deployment. source
- Desktop platforms
- The desktop app page lists Linux, macOS, and Windows assets for amd64 and arm64; it also says releases are not signed with Apple Developer or Windows code-signing certificates. source
- Footprint
- The gateway is described as an approximately 10.8 MB static binary designed to scale horizontally with Kubernetes HPA. source
- Kubernetes
- A Kubernetes Operator provides custom resources for managing gateways, agents, MCP servers, and chat orchestrators. source
- Kubernetes
- A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators declaratively as cluster resources. source
- License
- The documentation states that Inference Gateway is released under the Apache-2.0 license. source
- License and support
- The project is released under Apache-2.0, with GitHub Discussions for questions and an issue tracker for bugs and feature requests. source
- MCP integration
- MCP support discovers tools from connected servers, can connect to multiple servers, and executes tool calls through the gateway. source
- MCP integration
- It can discover tools from connected MCP servers and execute tool calls server-side. source
- Observability
- The gateway documents Prometheus-compatible metrics, OpenTelemetry instrumentation, structured JSON logs, and Grafana, Loki, and Jaeger examples. source
- Observability
- Built-in observability includes Prometheus metrics, OTLP tracing, structured JSON logs, and reference Grafana dashboards. source
- Privacy
- The product page states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped. source
- Privacy
- The project states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped. source
- Product
- Inference Gateway is an open-source, cloud-native proxy that presents supported hosted and local LLM providers through one OpenAI-compatible API. source
- Provider coverage
- The supported providers page lists OpenAI, Anthropic, Groq, Cohere, Ollama, Ollama Cloud, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, NVIDIA, llama.cpp, Z-AI, and ElevenLabs. source
- Provider coverage
- The documentation lists providers including OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp. source
- Provider switching
- The gateway lets applications switch providers by changing configuration, without an application redeploy, and can route model aliases to pools of upstream deployments. source
- Purpose
- Inference Gateway is an open-source, cloud-native proxy that provides one OpenAI-compatible API for multiple hosted and local LLM providers. source
- Routing
- The gateway supports model routing so a stable model alias can point to a pool of upstream deployments. source
Checked against the maker's pages on 10 Oct 2026.