# Self-Hosted AI Gateway: One API for Four Providers

> Self-hosted AI gateway built into Temps: one OpenAI-compatible endpoint for OpenAI, Anthropic, Gemini and Grok, with per-tag cost tracking and BYOK.

David Viejo · Published 2026-03-11 · Updated 2026-10-10 · 12 min read

Canonical: https://temps.sh/blog/ai-gateway-self-hosted-paas

## What Is a Self-Hosted AI Gateway?

A self-hosted AI gateway is a proxy on your own infrastructure that puts one endpoint in front of several LLM providers; [how to build an AI gateway for multiple LLM providers](https://temps.sh/blog/how-to-build-ai-gateway-multiple-llm-providers) explains the pattern in general.

**Temps ships an AI Gateway as a built-in feature** — no separate service to deploy, no additional bill. It runs inside the same Rust binary that handles your deployments, analytics, and error tracking.

> **TL;DR:** The Temps AI Gateway is an OpenAI-compatible proxy built into the Temps self-hosted PaaS. It routes requests to Anthropic (Claude), OpenAI (GPT/o-series), Google Gemini, and xAI (Grok) through one endpoint, with per-request token usage logged and spend estimated from a built-in price table. Provider API keys are encrypted with AES-256-GCM. Temps is licensed MIT / Apache 2.0 and free to self-host.

---

## The Problem: AI Operational Sprawl

AI spend keeps growing: respondents to [CloudZero's March 2025 survey](https://www.cloudzero.com/state-of-ai-costs/) expected average monthly AI spend to reach $85,521 in 2025. And most teams can't answer "which model is burning through our budget?"

Here's what happens without a gateway:

- **4+ API keys** scattered across env vars, Vault secrets, and team Slack DMs
- **4+ billing portals** with different invoicing cycles
- **No cross-provider cost view** — you can't answer "how much did our AI features cost last month?"
- **No per-user or per-feature attribution** — was it the chatbot or the code reviewer that burned $3,000?
- **Key rotation nightmares** — changing one key means updating 12 services

---

## Temps AI Gateway vs Running LiteLLM or Helicone

|                                 | Temps AI Gateway                                            | Running LiteLLM or Helicone yourself                                        |
| ------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Deployment**                  | Built into your existing Temps binary — no extra service    | A separate service to deploy, upgrade and monitor, with its own data stores |
| **Fallbacks between providers** | No (retry in your client)                                   | Yes                                                                         |
| **Auth and dashboard**          | Same auth, dashboard, and observability as your deployments | Their own keys and dashboard                                                |

The core difference: Temps embeds the gateway in the same process as deployments, analytics, error tracking, and uptime monitoring. You don't run a separate service, don't pay a separate bill, and don't log in to a separate dashboard. The tradeoff is fewer gateway features: LiteLLM and Helicone both do provider fallbacks, which Temps doesn't. For the full comparison of LiteLLM, Portkey, Helicone and Kong, with licences and the infrastructure each needs, see [how to build an AI gateway for multiple LLM providers](https://temps.sh/blog/how-to-build-ai-gateway-multiple-llm-providers#how-do-the-top-open-source-ai-gateway-options-compare).

---

## How the Temps AI Gateway Works

The gateway exposes three OpenAI-compatible endpoints (full reference in the [AI gateway docs](https://temps.sh/docs/ai-gateway)):

```
POST /api/ai/v1/chat/completions   → Chat (Anthropic, OpenAI, Gemini, xAI)
POST /api/ai/v1/embeddings         → Embeddings (OpenAI)
GET  /api/ai/v1/models             → List available models
```

Every request flows through this pipeline:

1. **Authentication** — Bearer token from your Temps API key
2. **Model routing** — Model name prefix determines the provider (`claude-*` → Anthropic, `gpt-*`/`o1`/`o3`/`o4` → OpenAI, `gemini-*` → Google, `grok-*` → xAI)
3. **Request translation** — OpenAI-format request gets translated to each provider's native API format
4. **Upstream call** — The gateway makes the actual API call with the encrypted provider key
5. **Response normalization** — Provider response is normalized back to OpenAI format
6. **Usage logging** — Tokens, latency, model, provider, conversation ID and tags are recorded to a TimescaleDB hypertable; the console estimates spend from a built-in price table

The gateway adds one hop and a format translation; the upstream model call dominates request time.

### Supported Providers

| Provider                   | Models                                                                                             | Streaming |
| -------------------------- | -------------------------------------------------------------------------------------------------- | --------- |
| **Anthropic**              | Any `claude-*` model                                                                               | Yes       |
| **OpenAI**                 | Any `gpt-*`, `o1`, `o3`, `o4`, `chatgpt-*`, `codex-*`, `text-embedding-*`, `dall-e*`               | Yes       |
| **Google**                 | Any `gemini-*` model                                                                               | Yes       |
| **xAI**                    | Any `grok-*` model                                                                                 | Yes       |
| **Custom upstream (BYOK)** | `x-provider-base-url` with your own key; the model name must still match one of the prefixes above | Yes       |

Routing is by model name prefix — case-insensitive. Unknown prefixes return a `ModelNotFound` error.

---

## One-Line Integration: Just Change the Base URL

If your code already calls OpenAI, switching to the Temps gateway takes one line:

```python
import openai

client = openai.OpenAI(
    api_key="tk_your_temps_api_key",
    base_url="https://your-temps.example.com/api/ai/v1",
)

# Routes to Anthropic automatically based on model prefix
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Explain TCP vs UDP"}],
)
```

Same code, same SDK, same types. Works with any OpenAI-compatible SDK — Python, TypeScript, Go, Rust, curl.

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "tk_abc123",
  baseURL: "https://your-temps.example.com/api/ai/v1",
});

// Four providers, one endpoint, one API key
const anthropic = await client.chat.completions.create({
  model: "claude-sonnet-4-6",
  messages: [{ role: "user", content: "Hello" }],
});

const openaiModel = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
});

const gemini = await client.chat.completions.create({
  model: "gemini-2.5-flash",
  messages: [{ role: "user", content: "Hello" }],
});
```

### Bring Your Own Key (BYOK)

Don't want to store provider keys in Temps? Pass them per-request via `X-Provider-Api-Key` header:

```bash
curl https://your-temps.example.com/api/ai/v1/chat/completions \
  -H "Authorization: Bearer tk_your_temps_key" \
  -H "X-Provider-Api-Key: sk-ant-your-anthropic-key" \
  -d '{"model": "claude-sonnet-4-6", "messages": [...]}'
```

BYOK keys are ephemeral — used for the request and never stored. System-configured keys are encrypted at rest with AES-256-GCM (via the `aes-gcm` Rust crate, same encryption layer used for deployment environment variables).

---

## Usage Analytics: Know Where Every Token Goes

Every request through the gateway gets logged with full metadata:

- **Provider and model** — which model handled the request
- **Token counts** — input and output tokens
- **Latency** — end-to-end response time in milliseconds
- **Cost** — estimated in the console by multiplying token counts by the built-in price table (`GET /api/ai/pricing`); the stored cost field currently reads zero, so compute cost the same way if you query the API
- **Credential type** — was this a system key (admin-configured) or BYOK?
- **Streaming flag** — was the response streamed?
- **Tags** — arbitrary labels like `team:platform`, `feature:chatbot`, `env:staging`

The dashboard provides:

- **Summary cards** — total requests, tokens, cost, average latency, error rate
- **Timeseries charts** — requests and tokens over time with hourly/daily bucketing
- **Per-model breakdown** — which models cost the most, which are fastest
- **Per-provider view** — compare Anthropic vs OpenAI vs Gemini at a glance

### Tag-Based Cost Allocation

Pass tags via the `x-tags` header to attribute costs to teams, features, or environments:

```bash
curl ... \
  -H "x-tags: team:ml-ops, feature:code-review, env:production" \
  -d '{"model": "claude-sonnet-4-6", ...}'
```

Then filter your analytics by any tag combination. Finally answer: "How much does the code review feature cost us per month?"

---

## OpenTelemetry Integration: Trace Every AI Call

Temps has built-in OTLP ingest (`/api/otel`): send OpenTelemetry GenAI spans from your app and they appear in the AI Activity view. Combine the AI Gateway with OTel GenAI semantic conventions for end-to-end traces of every AI interaction:

- Which user triggered the request
- What the input and output messages were
- How long the model took to respond
- Token usage and estimated cost
- Thinking blocks and tool calls (for Anthropic Claude)

The AI Activity tab in your project dashboard shows these traces as a conversation view — system prompt, user messages, assistant responses, tool calls, and reasoning/thinking blocks rendered inline.

```python
from opentelemetry import trace

tracer = trace.get_tracer("my-ai-app")

with tracer.start_as_current_span("chat claude-sonnet-4-6"):
    response = client.chat.completions.create(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": "..."}],
    )
```

---

## Setting Up the Temps AI Gateway: 3 Steps

### Step 1: Configure Provider Keys

In the Temps dashboard, go to **AI Gateway** and add your provider API keys. Each key is encrypted with AES-256-GCM before being written to the database. The dashboard validates the key format and tests connectivity before saving.

No CLI command required for this step.

### Step 2: Get Your Temps API Key

```bash
bunx @temps-sdk/cli api-key create --name "ai-gateway" --role admin
# → tk_abc123...
```

This is the same API key you use for deployments and the CLI — no new credentials to manage.

### Step 3: Point Your Code at the Gateway

```python
import openai

client = openai.OpenAI(
    api_key="tk_abc123",
    base_url="https://your-temps.example.com/api/ai/v1",
)
```

That's it. Four providers, one endpoint, one API key.

---

## What Teams Build With This

**AI-powered support bots** — Route customer queries to the cheapest model that meets quality thresholds. Track cost per conversation.

**Code review pipelines** — Use Claude for code analysis, GPT for documentation generation, Gemini for test generation. See which step costs the most.

**AI coding agents** — Route the agent's model calls through the gateway and run the code it writes on the same server, using [self-hosted sandboxes instead of a per-second managed service](https://temps.sh/blog/best-e2b-alternatives-ai-sandboxes-2026).

**RAG applications** — Embeddings through OpenAI, retrieval-augmented generation through Anthropic. One endpoint, unified cost tracking.

**Multi-model comparison** — A/B test models by sending identical requests to different providers and comparing quality vs cost vs latency.

**Internal AI tools** — Give every team access to AI without sharing raw API keys. Track usage per team via tags.

---

## Frequently Asked Questions

### Can I use the gateway without storing my API keys in Temps?

Yes. Use the `X-Provider-Api-Key` header to pass your key per-request (BYOK mode). It's used for that request only and never stored. System-configured keys are encrypted with AES-256-GCM.

### Does streaming work through the gateway?

Yes. All four providers support streaming via `"stream": true`. The gateway translates SSE chunks between provider formats: Anthropic's `content_block_delta` events, which carry the text, become OpenAI `delta` chunks, and its `message_delta` event, which carries the stop reason and usage, becomes the final chunk's `finish_reason` and token counts.

### How is the AI Gateway priced?

The AI Gateway is included in Temps at no extra charge. You pay your provider API costs directly (OpenAI, Anthropic, etc.) — Temps adds zero markup. It will also be included with Temps Cloud, a hosted control plane for your own servers, coming soon from $29/month. Self-hosting is free under the MIT / Apache 2.0 licenses.

### Can I see the actual conversation messages in traces?

Yes. Instrument your app with OpenTelemetry GenAI semantic conventions and the AI Activity tab shows full conversation threads — system prompts, user messages, assistant responses, tool calls, and thinking blocks.

### Which AI providers does Temps support?

Temps routes by model name prefix: `claude-*` → Anthropic, `gpt-*`/`o1`/`o3`/`o4`/`text-embedding-*` → OpenAI, `gemini-*` → Google, `grok-*` → xAI. With BYOK you can also point a request at a custom OpenAI-compatible base URL via `x-provider-base-url`, as long as the model name uses one of those prefixes.

---

## Start Using It Today

The AI Gateway ships with the latest Temps release. If you're already running Temps, update to the latest version and configure your provider keys in the dashboard.

If you're new to Temps, install it on a Linux server (see the [quickstart](https://temps.sh/docs/quickstart)):

```bash
curl -fsSL https://temps.sh/deploy.sh | bash
```

One binary. Deployments, analytics, error tracking, monitoring, and a built-in AI gateway. No SaaS sprawl required. MIT / Apache 2.0, free to self-host.

**Links:**

- [AI Gateway documentation](https://temps.sh/docs/ai-gateway)
- [OpenTelemetry tracing guide](https://temps.sh/blog/how-to-set-up-opentelemetry-tracing)
- [Temps GitHub](https://github.com/gotempsh/temps)

---

---

Source: https://temps.sh/blog/ai-gateway-self-hosted-paas · Index of all pages: https://temps.sh/llms.txt