Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

Inference Gateway

docs.inference-gateway.com

  • Free plan
  • Windows
  • macOS
  • Linux
  • Web
  • API
  • Self-hosted

What Inference Gateway is

Inference Gateway is an open-source, cloud-native proxy that gives applications one OpenAI-compatible API for hosted and local LLM providers. Its provider list includes OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Google, Mistral, NVIDIA, and others. Applications can change providers through configuration without a redeploy, while model aliases can route requests to pools of upstream deployments. The gateway can discover tools on connected MCP servers and execute calls through the gateway. Agent-to-Agent support enables agents to discover capabilities, delegate tasks, and stream results; agents can be defined in YAML, with a CLI that generates Go, Rust, or TypeScript projects. A Kubernetes Operator offers custom resources for gateways, agents, MCP servers, and chat orchestrators. Documentation covers OIDC authentication, Prometheus-compatible metrics, OpenTelemetry, and structured JSON logs, with Grafana, Loki, and Jaeger examples. Docker and Kubernetes deployment are documented. The project says it can be self-hosted, including in air-gapped environments, and has no analytics or telemetry phoning home. It is released under Apache-2.0.

Who it suits

It suits teams that want one API for multiple hosted and local LLM providers, with configuration-based switching and model routing. Operators who need self-hosting, MCP connections, or Kubernetes management can also consider it.

Good to know, and what to watch

  • One OpenAI-compatible API covers hosted and local providers.
  • Provider changes do not require an application redeploy.
  • Supports MCP tools and agent coordination.
  • Documents Docker and Kubernetes deployment.
  • Released under Apache-2.0.
  • Desktop releases are not signed with Apple or Windows certificates.

The verdict

Inference Gateway offers a broad integration surface for teams that want to route among multiple LLM providers and operate the gateway themselves. Review its deployment and signing details against your environment before adopting it.

What the maker publishes

Agent coordination
Its Agent-to-Agent support lets agents discover capabilities, delegate tasks, and stream results within a conversation. source
Agent workflows
The product supports Agent-to-Agent coordination and describes defining agents in YAML with a CLI that generates Go, Rust, or TypeScript projects. source
Authentication
OIDC authentication validates JWTs from a configured identity provider, and the gateway documentation says the gateway needs no client secret for token validation. source
Authentication
Authentication supports OIDC identity providers, with JWT validation against the issuer’s JWKS. source
Deployment
The documentation provides a Docker image and describes running the gateway in Docker or Kubernetes. source
Deployment
The getting-started guide documents Docker, Docker Compose examples, and Kubernetes deployment. source
Desktop platforms
The desktop app page lists Linux, macOS, and Windows assets for amd64 and arm64; it also says releases are not signed with Apple Developer or Windows code-signing certificates. source
Footprint
The gateway is described as an approximately 10.8 MB static binary designed to scale horizontally with Kubernetes HPA. source
Kubernetes
A Kubernetes Operator provides custom resources for managing gateways, agents, MCP servers, and chat orchestrators. source
Kubernetes
A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators declaratively as cluster resources. source
License
The documentation states that Inference Gateway is released under the Apache-2.0 license. source
License and support
The project is released under Apache-2.0, with GitHub Discussions for questions and an issue tracker for bugs and feature requests. source
MCP integration
MCP support discovers tools from connected servers, can connect to multiple servers, and executes tool calls through the gateway. source
MCP integration
It can discover tools from connected MCP servers and execute tool calls server-side. source
Observability
The gateway documents Prometheus-compatible metrics, OpenTelemetry instrumentation, structured JSON logs, and Grafana, Loki, and Jaeger examples. source
Observability
Built-in observability includes Prometheus metrics, OTLP tracing, structured JSON logs, and reference Grafana dashboards. source
Privacy
The product page states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped. source
Privacy
The project states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped. source
Product
Inference Gateway is an open-source, cloud-native proxy that presents supported hosted and local LLM providers through one OpenAI-compatible API. source
Provider coverage
The supported providers page lists OpenAI, Anthropic, Groq, Cohere, Ollama, Ollama Cloud, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, NVIDIA, llama.cpp, Z-AI, and ElevenLabs. source
Provider coverage
The documentation lists providers including OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp. source
Provider switching
The gateway lets applications switch providers by changing configuration, without an application redeploy, and can route model aliases to pools of upstream deployments. source
Purpose
Inference Gateway is an open-source, cloud-native proxy that provides one OpenAI-compatible API for multiple hosted and local LLM providers. source
Routing
The gateway supports model routing so a stable model alias can point to a pool of upstream deployments. source

Checked against the maker's pages on 10 Oct 2026.