LLM Gateway Architecture: Centralized Routing, Budget Control, and Provider Fallbacks

Most SaaS teams ship their first AI feature the direct way. Application code calls the provider SDK, the API key sits in an environment variable, and everything works. Then a second feature ships, then a third, and suddenly budget checks, retry logic, and model selection live in five places with five slightly different behaviors. LLM gateway architecture is the structural answer: a single centralized service that sits between your application code and the model providers, and owns routing, budget enforcement, failover, and caching in one place. This post covers what a gateway should handle, how to build a lightweight one on a Symfony HTTP kernel, and how consumers in Symfony and Next.js talk to it.

We have written before about token budgets and cost control and about fallback patterns for provider outages as application-level patterns. This post is about the infrastructure pattern that makes those concerns disappear from application code entirely.

Why LLM gateway architecture beats scattered provider calls

The direct-call approach fails in a predictable order. First cost visibility goes. When six features call two providers with different models, nobody can answer the question "which tenant cost us the most last month" without stitching together provider dashboards that were never designed for per-customer attribution. Then consistency goes. One feature retries on rate limits, another does not. One team pinned an old model version, another follows the latest. Finally, incidents get expensive: when a provider has a bad afternoon, you patch failover logic into every call site under pressure instead of flipping one switch.

A gateway inverts the ownership. Application code makes one kind of request to one internal endpoint: here is the task type, the tenant, the prompt. The gateway decides which provider and model serve it, whether the tenant still has budget, whether a cached answer is good enough, and what to do when the primary provider returns errors. Every policy lives in one deployable service with one config file, and changing a model or adding a provider is a gateway change, not a hunt through repositories.

There is a build-or-buy question here. Commercial and open source gateways exist, and for some teams they are the right call. But the pattern is small enough that owning it is realistic, and owning it means budget logic that matches your billing model exactly instead of approximately. The decision follows the same logic as any tech stack strategy call: buy when the problem is generic, build when the policy is your product.

Per-tenant token budgets that actually enforce something

Budget control is the feature that usually justifies the gateway. Provider-side spend limits protect you from a global blowout, but they know nothing about your tenants. The gateway does, because every request carries a tenant identifier.

The enforcement model that works in practice has three layers. A hard limit is the ceiling: when a tenant's token consumption for the billing period crosses it, the gateway rejects further requests with a clear, machine-readable error the application can translate into UI. A soft limit sits below it, typically at 80 percent, and triggers alerts to your team and optionally to the customer, so nobody discovers the ceiling by hitting it. And a monthly reset ties consumption windows to your billing cycle rather than the calendar, which matters as soon as tenants have different renewal dates.

The bookkeeping is less trivial than it sounds. Token counts are only known after the response arrives, so the gateway records actual usage post-hoc while making the admission decision on the running total. That leaves a small window where parallel requests from the same tenant can overshoot the ceiling slightly. For token budgets this is fine. Reserve pessimistic locking for cases where overshoot has contractual consequences, and accept eventual consistency everywhere else. A PostgreSQL table keyed on tenant and billing period, updated with an atomic increment, handles this at any scale a mid-sized SaaS will see.

Routing: the right model for each request, not one model for everything

Once every request flows through one service, routing stops being a hardcoded constant and becomes policy. The gateway classifies requests by task type, declared by the caller, and maps each type to a model based on three inputs: capability, latency, and cost.

The practical wins are unglamorous and large. Short classification and extraction tasks route to small, cheap, fast models. Long-form generation and complex reasoning route to frontier models like GPT-4o or Claude Opus. A summarization task that tolerates two seconds of extra latency routes to whichever qualified model is cheapest this quarter. When a provider ships a better model at half the price, you update one mapping and every feature benefits the same afternoon.

Routing policy belongs in versioned configuration, not code. A YAML map from task type to an ordered list of provider and model pairs, with per-entry timeouts and cost metadata, is enough. The ordering doubles as your failover chain, which is the next concern.

Failover on 429 and 5xx, in one place

Provider failure handling is where duplicated call sites hurt the most, and where centralization pays off immediately. The gateway watches response codes and applies one consistent policy. A 429 means backoff with jitter and, if pressure continues, spillover to the next model in the chain. A 5xx or a timeout means retry once, then fail over. Repeated failures within a short window trip a circuit breaker that routes all traffic for that provider straight to the fallback until a probe request succeeds.

Because the routing table already defines an ordered list per task type, failover is just walking the list. The important design decision is that the caller never sees any of it. The application asked for a summarization; whether it was served by the primary model or a secondary provider mid-incident is a gateway log line, not an application concern. Pair providers deliberately so the second entry in each chain is hosted by a different vendor, otherwise a regional incident takes out the whole list at once.

Semantic caching with pgvector

Exact-match response caching helps less with LLM traffic than teams hope, because prompts embed user-specific content and rarely repeat byte-for-byte. Semantic caching fixes that: embed the incoming prompt, search for previously answered prompts within a similarity threshold, and return the stored answer when one is close enough.

Since the gateway already runs beside PostgreSQL for budget tracking, pgvector makes this cheap to add. Store the embedding, the response, the model that produced it, and a TTL. On each request, one indexed similarity query decides between a cache hit that costs a fraction of a cent and a model call that costs a hundred times more. Scope cache entries per tenant unless prompts are genuinely tenant-neutral, and keep the similarity threshold conservative: a wrong-but-similar answer damages trust in a way a cache miss never does. FAQ-style features routinely see meaningful hit rates here, and the technique compounds with provider-side prompt caching rather than replacing it.

Logging for cost attribution and debugging

Every request through the gateway produces one structured log record: tenant, task type, model requested, model actually used, token counts in and out, latency, cache hit or miss, failover events, and computed cost. This single stream answers the questions that are otherwise painful. Which tenants are unprofitable at their current plan. Which feature drives the token bill. Whether last Tuesday's latency spike was your code or the provider. What the real cache hit rate is.

Retention needs a policy from day one, because prompts contain customer data. Log token counts and metadata indefinitely, but store prompt and response bodies short-term, redacted, or not at all, depending on what your data processing agreements allow.

A lightweight Symfony implementation

The gateway does not need a framework's full weight, and it should not have one. A slim Symfony HTTP kernel with a handful of routes keeps cold paths short while giving you the HttpClient component, which already speaks retries, timeouts, and streaming.

The shape of the service: a POST endpoint accepting task type, tenant ID, messages, and options. Middleware authenticates the internal caller, loads the tenant budget row, and rejects over-limit requests before any provider is contacted. Redis handles rate limiting per tenant and the circuit breaker state, both natural fits for its atomic counters and TTLs. PostgreSQL holds budget ledgers, the pgvector cache, and the request log. The provider adapters are thin classes mapping the internal request format onto each vendor's API, which is also the seam where you normalize streaming.

Define the API in an OpenAPI spec and treat it as the contract. Generate or hand-write two small client libraries against it: a Symfony HTTP client wrapper for your PHP services, and a typed fetch wrapper for Next.js server routes. Keep the clients dumb by design. Every piece of intelligence that creeps into a client eventually disagrees with the gateway, and you are back to scattered policy. The whole service lands at a few thousand lines, which is the point: this is the kind of focused infrastructure component we build in custom software development engagements in weeks, not quarters.

When you do not need a gateway

One feature, one provider, one team: call the provider directly and keep the fallback logic in application code. The gateway earns its operational cost when at least two of these are true: multiple features consume LLMs, you bill or budget per tenant, you use more than one provider, or model choice changes often enough that hardcoding it hurts. Adopting the pattern too early adds a network hop and a service to operate for policy you do not yet have. Adopting it too late means migrating five call sites that each grew their own behavior. The second migration is worse. If you are unsure which side of the line you are on, an architecture review of your current LLM call sites usually makes the answer obvious.

Where to start

Build the gateway around the concern that hurts today. If that is cost, ship budget enforcement and logging first and add routing later. If it is reliability, start with the routing table and failover and bolt on budgets afterwards. The architecture supports incremental adoption because every capability hangs off the same chokepoint.

If you are weighing an LLM gateway for your SaaS, or you have five direct provider integrations and a token bill nobody can explain, we can help you design and build the right-sized version. Write to hello@wolf-tech.io or find us at wolf-tech.io.