AI Egress Gateway
A self-hosted proxy that centralizes AI provider keys, rate limits, and enforces hard cost caps
The problem it solves
When you have multiple apps calling AI providers, provider API keys end up in multiple places. Cost visibility is fragmented across provider dashboards. Rate limiting is either absent or handled ad hoc in each app. The gateway centralizes all of this: one place where keys live, one place where costs are metered, one place where rate limits are enforced.
How it works
Each app routes its LLM requests through the gateway instead of calling providers directly. The gateway verifies the incoming JWT against the Klass Accounts JWKS endpoint, applies the per-app rate limit configuration, checks the current period spend against the configured cost cap, proxies the request to the correct provider, and records the response tokens for usage metering. SSE streaming responses are passed through to the caller without buffering.
Fail-closed cost caps
The cost cap design is deliberate. When an app exceeds its configured per-period spend limit, requests fail with a clear error - they don't succeed silently. This is different from how most rate limiting works. The reasoning is that unexpected AI costs are a real operational risk, and failing closed makes overspend impossible at the cost of service interruption. The cap, period, and behavior are all configurable per app.
What's left
The core proxy, JWT verification, rate limiting, and cost metering are working. What's still in progress is the admin API for managing app registrations and viewing usage reports, and the Gemini adapter (Anthropic and OpenAI adapters are complete). The gateway is being designed to eventually handle all AI egress across my projects.
Stack
Want something like this built?
This is the kind of system I build for clients. Tell me what you're trying to do, and I'll give you a straight answer on whether AI is the right tool.
Book a 30-minute call