An open-source, cloud-native, high-performance gateway unifying multiple LLM providers
📖 Documentation · 🚀 Getting Started · 💬 Discussions · 🐛 Issues
Inference Gateway is a proxy server that provides a unified API to interact with multiple large language model (LLM) providers — from local solutions like Ollama to major cloud providers like OpenAI, Anthropic, Groq, Cohere, Cloudflare, and DeepSeek.
Stop managing multiple SDKs and API keys. Route all your LLM traffic through a single, production-ready gateway.
# One endpoint. Every provider.
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Hello!"}]}'| Feature | Description |
|---|---|
| 🔀 Unified API | One OpenAI-compatible endpoint for all LLM providers |
| 🔌 MCP Integration | Native Model Context Protocol support for automatic tool discovery |
| 🤖 A2A Protocol | Agent-to-Agent coordination across specialized agents |
| 🌊 Streaming | Real-time token streaming from all supported providers |
| ☸️ Kubernetes Ready | First-class K8s support with Operator and HPA scaling |
| 📊 Observability | OpenTelemetry integration for monitoring and tracing |
| 🔒 Privacy First | Self-hosted, zero data collection, MIT licensed |
| 🌿 Lightweight | ~10.8MB binary with minimal resource footprint |
| Repository | Description |
|---|---|
| inference-gateway | The core gateway server |
| operator | Kubernetes Operator for lifecycle management |
| cli | Agentic CLI assistant with project context awareness |
| adl-cli | Scaffold and manage A2A-powered enterprise agents |
| Repository | Language |
|---|---|
| sdk | Go |
| rust-sdk | Rust |
| Repository | Description |
|---|---|
| adk | Agent Development Kit for building A2A-compatible agents |
| google-calendar-agent | Google Calendar scheduling & automation |
| browser-agent | Browser automation via Playwright |
| documentation-agent | Context7-style documentation access for agents |
# Run with Docker
docker run -p 8080:8080 \
-e OPENAI_API_KEY=your-key \
ghcr.io/inference-gateway/inference-gateway:latest
# Or install the CLI
curl -fsSL https://raw.githubusercontent.com/inference-gateway/cli/main/install.sh | bash
infer init && infer chat👉 Full setup guide: docs.inference-gateway.com/getting-started
We welcome contributions of all kinds — bug reports, feature requests, documentation improvements, and code!
- ⭐ Star the main repo to show your support
- 🐛 Report bugs via GitHub Issues
- 💬 Join discussions in GitHub Discussions
- 🔧 Submit PRs — see
CONTRIBUTING.mdin each repository
Released under the MIT License · Built with ❤️ in Go