Anatomy of a governed LLM request

“Governed by default” is easy to say and expensive to mean. This post walks one chat completion through Eparch’s gateway, stage by stage, to show what the phrase buys you in practice.

The setup: an incident-triage agent wants a completion. It sends an OpenAI-compatible POST /v1/chat/completions to the Eparch gateway — the same request shape it would send to any provider. Everything below happens before and after the model sees it.

1. Authentication

Every request carries a platform JWT — an RS256 token minted by Eparch’s identity service, not a pass-through of whatever your IdP issued. Identity brokers the login (WorkOS AuthKit for enterprise SSO, or built-in email/password auth in local mode) and then issues the platform’s own token, so downstream services verify one signature against one JWKS endpoint. No valid token, no pipeline: the request dies here with a 401.

2. Policy

Next, an Open Policy Agent interceptor evaluates the request against your policy: is this subject allowed to perform this action on this resource, in this tenant? Policy is written in Rego and enforced uniformly — the same interceptor guards every service, so there is no service that “forgot” to check. A deny is logged and returned; the model is never consulted.

3. Rate limits

Redis-backed rate limiting is applied per tenant. This is the guard against a runaway agent loop or a misbehaving client saturating your GPUs — the failure mode every self-hosted inference deployment eventually meets.

4. Budgets

Then the budget check: every tenant has daily and monthly spend limits, tracked in Redis and accounted in the database. A request that would blow the budget is refused before the tokens are generated, not flagged in a report three weeks later. Cost control as a precondition, not a dashboard.

5. Semantic cache

The gateway keeps a semantic response cache. If a semantically equivalent prompt has been answered recently, the cached response is returned — the request never touches a GPU. On cache hit, total marginal cost: one embedding comparison.

6. Routing and fallback

A cache miss finally reaches model routing. Rules select a backend by task and tier, with fallback if the primary is down or over capacity:

routes:
  - match: { task: 'chat', tier: 'fast' }
    model: llama-3.3-70b
    fallback: qwen-2.5-72b
  - default: llama-3.3-70b

The backend is whatever OpenAI-compatible endpoint you registered — vLLM on your own GPUs, Ollama on a workstation, or a hosted provider if your policy allows one. The gateway holds an allow-list of models, so an agent can’t quietly request one you haven’t approved.

7. Audit and accounting

The response streams back to the agent, and the request’s full story is written down: tenant, actor, action, model, token counts, cost, and the policy decision, in an immutable per-tenant audit log.

{
  "tenant": "9f2c…",
  "actor": "agent:incident-triage",
  "action": "llm.chat",
  "model": "llama-3.3-70b",
  "tokens": { "prompt": 1204, "completion": 312 },
  "costUsd": 0.0041,
  "decision": "allow"
}

This is the artifact your security review actually wants: not a promise that requests are governed, but a queryable record that each one was.

Why a pipeline and not a library

Every stage above could be a library your agents are supposed to call. In practice, “supposed to” is how governance fails — one team forgets the budget check, one script bypasses the allow-list, and the audit trail has holes exactly where the incidents are. Eparch makes the pipeline the only door: agents speak the OpenAI-compatible API to the gateway, and the gateway does the governing. The agent code stays simple; the guarantees stay uniform.

The whole pipeline is open source and runs on your hardware. See it locally in about two minutes:

git clone https://github.com/eparchai/eparch && cd eparch && make quickstart

Then read the getting-started guide to wire it to a real model.