How to Build an AI Gateway for Multiple LLMs

An AI gateway is a reverse proxy between your application and LLM providers that normalizes APIs, tracks per-token costs, can fail over between providers, and eliminates SDK sprawl across multiple providers. You can build one yourself in about 2,000–3,000 lines of well-tested code, run an open-source gateway like LiteLLM, Portkey or Helicone, or use the OpenAI-compatible gateway built into a platform you already run, such as Temps.

This guide walks through the architecture, a working DIY implementation, and a comparison of the open-source options.

TL;DR: An AI gateway routes requests from one OpenAI-compatible endpoint to multiple LLM providers (OpenAI, Anthropic, Gemini, xAI), tracks costs by feature and team, and can fail over between providers. Respondents to CloudZero's March 2025 survey expected average monthly AI spend to rise to $85,521 in 2025, yet most teams can't attribute that spend to specific features. A gateway solves both routing and visibility in one layer.

What Is an AI Gateway?

An AI gateway is a reverse proxy that sits between your application code and LLM providers. The pattern centralizes authentication, request routing, usage tracking, and response normalization into a single layer.

Think of it like an API gateway (Kong, Envoy, or AWS API Gateway) but purpose-built for LLM traffic. A traditional API gateway routes HTTP requests. An AI gateway understands tokens, models, streaming chunks, and cost-per-million-token pricing.

How It Works

Here's the basic flow:

┌─────────────┐     ┌──────────────────┐     ┌───────────────┐
│             │     │                  │     │   OpenAI      │
│  Your App   │────▶│   AI Gateway     │────▶│   Anthropic   │
│  (one SDK)  │◀────│                  │◀────│   Gemini      │
│             │     │  - Auth          │     │   xAI         │
└─────────────┘     │  - Route         │     └───────────────┘
                    │  - Translate     │
                    │  - Log           │
                    │  - Cache         │
                    └──────────────────┘

Your app sends every request in one format — typically OpenAI's /chat/completions schema. The gateway inspects the model field, routes to the correct provider, translates the request into that provider's native API format, translates the response back, and logs everything along the way.

How It Differs from a Regular API Gateway

That last row matters. A single /chat/completions endpoint might fan out to four different providers, each with its own API key, rate limit, and error format. A regular API gateway doesn't handle that.

Why Do You Need One Even for a Single Provider?

Even teams using just OpenAI benefit from a gateway layer. The gateway isn't just about multi-provider routing — it's about operational visibility.

Here's what a gateway gives you beyond multi-provider support:

Cost Tracking per Feature, User, and Team

Without a gateway, you see one number on your OpenAI bill: total spend. With a gateway, every request carries metadata — which user triggered it, which feature it belongs to, which environment it ran in. You can finally answer "how much does our chatbot cost per conversation?"

Rate Limiting at the Application Level

Provider rate limits protect the provider. They don't protect you from a single user burning through your entire quota. A gateway lets you set limits per user, per team, or per feature — before the request ever hits the provider.

Caching Identical Prompts

If 500 users ask "what's your refund policy?" within an hour, that's 500 identical API calls. A gateway can cache responses by prompt hash and serve identical answers from cache; how much that saves depends on how repetitive your prompts are.

Audit Logging for Compliance

SOC 2 and HIPAA require logging access to sensitive data. If your app sends customer data to an LLM, you need an immutable record of what was sent, to which model, and when. A gateway creates this audit trail automatically.

Key Rotation Without Code Changes

Rotating an API key should be a config change, not a deployment. With a gateway, you update the key in one place. Every service that routes through the gateway picks up the new key immediately. No redeployment, no downtime.

Provider Failover

OpenAI has had significant outages. When your primary provider goes down, a gateway can automatically retry the request with a fallback model — say, Anthropic Claude when GPT-4.1 is unavailable. Your users never notice.

What Is the OpenAI-Compatible Interface Pattern?

OpenAI's /chat/completions format has become the de facto standard for LLM APIs: most model hosts, open-source servers and SDKs accept it. That convergence makes the "OpenAI-compatible" pattern the smartest foundation for any AI gateway.

Here's the core schema every developer already knows:

{
  "model": "gpt-4.1",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain TCP vs UDP."}
  ],
  "temperature": 0.7,
  "max_tokens": 1024,
  "stream": true
}

Anthropic, Google, xAI, Mistral, and dozens of open-source model hosts all have different native formats. But if your gateway accepts this one schema and translates under the hood, all existing code works. Every OpenAI SDK in Python, TypeScript, Go, or Rust connects by changing a single line: the base URL.

That's the key insight. Don't invent a new API format. Ride the ecosystem.

Why Not Just Use Each Provider's SDK?

You could. But consider what happens at scale:

• Your Python service uses openai and anthropic SDKs
• Your TypeScript service uses @anthropic-ai/sdk and @google/generative-ai
• Your Go service uses its own HTTP clients
• Each has different error types, retry logic, and streaming implementations

Now multiply by 10 services. That's 20–40 SDK dependencies with different versioning, different auth patterns, and different response shapes. A gateway collapses all of that into one client library and one response format.

How Do You Build a Basic AI Gateway From Scratch?

Building a minimal AI gateway is more work than most teams expect when they start: routing, per-provider request translation, streaming, retries and cost tracking all have to be written and maintained. Here's a walkthrough of the core pieces using Python and FastAPI. The snippets are illustrative excerpts, not one complete runnable file.

Step 1: Accept OpenAI-Format Requests

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()

PROVIDER_KEYS = {
    "openai": "sk-...",
    "anthropic": "sk-ant-...",
    "google": "AIza...",
}

def detect_provider(model: str) -> str:
    if model.startswith(("gpt-", "o1", "o3", "o4")):
        return "openai"
    elif model.startswith("claude-"):
        return "anthropic"
    elif model.startswith("gemini-"):
        return "google"
    raise ValueError(f"Unknown model: {model}")

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    body = await request.json()
    provider = detect_provider(body["model"])

    if body.get("stream"):
        return StreamingResponse(
            stream_response(provider, body),
            media_type="text/event-stream",
        )
    return await sync_response(provider, body)

That's the skeleton. stream_response() and sync_response() are left for you to implement: each translates the request for the chosen provider (Step 2), calls it with httpx, and converts the reply back to OpenAI format (add import json if you parse streamed chunks). Now comes the hard part.

Step 2: Translate Requests to Provider-Native Formats

OpenAI and Anthropic have fundamentally different request schemas. Here's the translation:

def translate_to_anthropic(body: dict) -> dict:
    """Convert OpenAI chat format to Anthropic Messages API."""
    messages = body["messages"]
    system = None
    converted = []

    for msg in messages:
        if msg["role"] == "system":
            system = msg["content"]
        else:
            converted.append({
                "role": msg["role"],
                "content": msg["content"],
            })

    payload = {
        "model": body["model"],
        "messages": converted,
        "max_tokens": body.get("max_tokens", 4096),
    }
    if system:
        payload["system"] = system
    if body.get("temperature") is not None:
        payload["temperature"] = body["temperature"]
    if body.get("stream"):
        payload["stream"] = True

    return payload

Notice the differences. Anthropic pulls system out of the messages array and into a top-level field. max_tokens is required (not optional). Temperature ranges differ. Tool calling schemas are different. And we haven't even touched Gemini's format yet, which uses contents instead of messages and has its own parts structure.

Step 3: Handle Streaming (The Hard Part)

Streaming is where the complexity explodes. Each provider uses Server-Sent Events, but the chunk format is completely different.

OpenAI sends:

data: {"choices":[{"delta":{"content":"Hello"}}]}

Anthropic sends:

event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Hello"}}

You need to parse each provider's SSE stream and normalize the chunks back to OpenAI's delta format. Here's a simplified version for Anthropic:

async def normalize_anthropic_stream(response):
    async for line in response.aiter_lines():
        if not line.startswith("data: "):
            continue
        data = json.loads(line[6:])

        if data["type"] == "content_block_delta":
            yield f'data: {json.dumps({
                "choices": [{
                    "delta": {"content": data["delta"]["text"]},
                    "index": 0,
                }]
            })}\n\n'
        elif data["type"] == "message_stop":
            yield "data: [DONE]\n\n"

This is simplified. The real implementation needs to handle message_start (for model info and input token counts), content_block_start, tool use blocks, thinking blocks, and error events mid-stream. It also needs to compute output token counts from message_delta usage fields.

Step 4: Token Counting and Cost Calculation

MODEL_PRICING = {
    "gpt-4.1": {"input": 2.00, "output": 8.00},
    "gpt-4.1-mini": {"input": 0.40, "output": 1.60},
    "gpt-4.1-nano": {"input": 0.10, "output": 0.40},
    "claude-sonnet-4-6": {"input": 3.00, "output": 15.00},
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 0.60},
}  # Example prices per 1M tokens. Claude rows checked against Anthropic's pricing page in October 2026; verify the others and keep this in config, not code.

def calculate_cost(model, input_tokens, output_tokens):
    prices = MODEL_PRICING.get(model, {"input": 0, "output": 0})
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)

But here's the catch. For streaming responses, you don't get final token counts until the stream ends. You need to accumulate counts across chunks, handle cases where the provider doesn't report them, and fall back to tokenizer-based estimation.

Why This Gets Unwieldy Fast

What we've built so far handles the happy path. A production-ready gateway also needs:

• Retry logic with exponential backoff per provider
• Circuit breakers to stop hitting a failing provider
• Request/response validation against schemas
• Timeout handling (LLM requests routinely take 10–30 seconds)
• Concurrent request limits per provider
• Error normalization — each provider returns errors differently
• Tool calling translation between OpenAI and Anthropic formats
• Content filtering for PII or prompt injection detection
• Persistent logging to a database

You're looking at 2,000–3,000 lines of well-tested code before it's production-ready. Is that worth building when solid open-source options exist?

How Do the Top Open-Source AI Gateway Options Compare?

The open-source AI gateway space has matured quickly. Here's how the main self-hosted options compare (licences and features checked on each project's GitHub repository, October 2026).

LiteLLM

The most popular open-source option. Supports 100+ models, handles the translation layer well, and has spend tracking by key, team and tag plus retries and fallbacks. It's a Python service that needs its own deployment, a PostgreSQL database for keys and spend, optionally Redis for caching, and its own monitoring.

Portkey Gateway

Portkey's open-source gateway is MIT-licensed and self-hostable for free (npx @portkey-ai/gateway or Docker), with automatic retries, fallbacks and load balancing across providers. Organisation management and governance features belong to Portkey's paid platform, and the richest dashboards live in Portkey's cloud.

Helicone

Helicone started as an observability proxy and now ships an AI Gateway with routing and automatic fallbacks. It is Apache-2.0 and self-hostable via docker-compose, but the self-hosted stack brings several services of its own, including ClickHouse.

Kong AI Gateway

Enterprise-grade but enterprise-heavy, and now a commercial product: open-source Kong has stayed on the 3.9.x line (3.9.3, June 2026, per its GitHub releases), and AI Gateway 2.x configures models through an AI Model entity instead of the open-source ai-proxy plugin (Kong AI Gateway docs, checked October 2026). It runs on a Kong deployment, with the configuration management and operational overhead that comes with it. A good fit if you already pay for Kong.

Every option above solves the gateway problem. The real difference between them is operational: each self-hosted gateway is one more service to deploy, upgrade and monitor, unless it ships inside something you already run.

Using the Gateway Built Into Temps

If you already deploy on Temps, you don't need a separate gateway service: the same binary that runs your deployments exposes OpenAI-compatible endpoints (/api/ai/v1/chat/completions, /api/ai/v1/embeddings, /api/ai/v1/models) in front of OpenAI, Anthropic, Gemini and xAI. Point any OpenAI SDK at it by changing the base URL and API key; provider keys are stored encrypted on the server, or passed per request with x-provider-api-key (BYOK). Every request is logged with model, tokens, latency, conversation ID and x-tags labels, and the console estimates spend from a built-in price table, so cost shows up per feature in the dashboard.

What it doesn't do today: per-user, per-team or per-tag rate limits, or automatic failover to another provider on a 429 — handle retries and fallbacks in your client. The AI gateway docs are the API reference, and self-hosted AI gateway on your PaaS covers when the built-in gateway beats LiteLLM or Helicone. If agents are calling the gateway, the code they write can run in self-hosted sandboxes on the same host.

How Do You Track AI Costs and Attribute Them to Specific Features?

Cost visibility is the primary reason teams adopt AI gateways. Per-token prices keep falling, but total spend usually climbs because usage grows faster. Knowing where the money goes matters more than the per-token price.

Whatever gateway you use, log each request with at least:

• Model and provider — which model handled this request
• Input/output tokens — exact counts from the provider response
• Latency — end-to-end time in milliseconds
• Cost — computed from a per-model price table
• Conversation ID — group multi-turn chats together
• Tags — arbitrary key:value labels

Tag-Based Attribution

Pass tags via headers to slice costs any way you want, then ask the dashboard a question like "how much did the code-review feature cost in production last month?" and get a number broken down by model. With the Temps gateway that is the x-tags header; the request and the filters are shown under tag-based cost allocation.

This is the kind of visibility that standalone billing dashboards from OpenAI or Anthropic don't provide. They show you total spend. A gateway shows you why you're spending.

Frequently Asked Questions

What is the latency overhead of an AI gateway?

A gateway adds one extra network hop plus request and response translation. That is small next to model inference, which takes hundreds of milliseconds to many seconds depending on the model and output length — but measure it on your own deployment rather than trusting a vendor's number, and run the gateway close to your apps.

Can I use an AI gateway with streaming responses?

Yes. Streaming is the most common mode for chat applications. The gateway translates Server-Sent Events between provider formats: Anthropic's content_block_delta events, which carry the text, become OpenAI's delta chunks, and its message_delta event, which carries the stop reason and usage, becomes the final chunk. You use "stream": true in your request body and the response streams through exactly like a direct OpenAI call.

How do I handle rate limits across multiple providers?

A gateway can give you two layers. First, application-level limits: cap requests per user, per team, or per feature before they reach a provider. Second, provider-aware handling: if OpenAI returns a 429, retry with backoff or fall back to another provider. LiteLLM, Portkey and Helicone offer fallbacks; the Temps gateway currently does neither per-user limits nor automatic fallback (x-tags only labels usage), so do retries and fallbacks in your client when you use it.

What's the difference between an AI gateway and a load balancer?

A load balancer distributes identical requests across identical backends. An AI gateway routes different models to different providers, translates between incompatible API formats, tracks per-token costs, and normalizes streaming chunk formats. They solve fundamentally different problems. You'd put a load balancer in front of multiple AI gateway instances, not use one instead.

Do I need an AI gateway if I only use one provider?

You don't need one, but you'll wish you had one. Single-provider teams still benefit from cost attribution per feature, key rotation without redeployment, caching identical prompts, audit logging, and rate limiting at the application level. And when you inevitably add a second provider, you won't need to refactor anything.

Start Routing to Multiple LLM Providers Today

The pattern is clear: a reverse proxy between your app and LLM providers eliminates SDK sprawl, centralizes cost tracking, and is the natural place for failover. You can build one yourself — the code above is a solid starting point. You can deploy LiteLLM or another open-source option and maintain it as a separate service. Or you can use Temps, which bundles the gateway alongside deployments, analytics, and error tracking in one binary.

If you're already running Temps, the AI gateway is built in. Configure your provider keys in the dashboard and start routing. If you're new, install it with:

curl -fsSL https://temps.sh/deploy.sh | bash

One binary. One endpoint. Every LLM provider your team needs.

Related guides:

• How to Set Up OpenTelemetry Tracing — add distributed tracing to your AI pipeline
• 7 Best Self-Hosted Deployment Platforms — platforms that include AI gateway alongside deployment