See also (wiki): ai-gateway · enterprise-agent-runtime-infrastructure · agentic-token-governance · llm-cost-attribution
Source files: LiteLLM GitHub PRs/issues and docs, Bifrost docs/pricing/repository snapshot, pierretokns/frankengate GitHub repository metadata and public project description, TensorZero docs/repository snapshot, Kong AI Gateway docs, Cloudflare AI Gateway docs, AWS Bedrock/AgentCore/CloudWatch/X-Ray docs, MLflow tracing/evaluation docs, Azure APIM Build 2026 release notes, Palo Alto Networks/Portkey press release, NVIDIA Perplexity technical blog, and sources/21-benchmarks/model-routing-meaningfulness-hse-robustness-raw.md.
Evidence Boundaries: Gateway product claims, acquisition values, pricing, latency, and customer-count statements are vendor/source-reported unless independently reproduced. OSS-vs-enterprise boundaries below are based on public docs and source-tree inspection, not a paid enterprise license test. FrankenGate’s inclusion is currently repository-metadata/project-description evidence; no source-level or local multi-pod conformance run has been completed. Coding-agent harness compatibility is config-contract evidence, not a full Claude Code/Codex behavioral regression run. Router-meaningfulness findings from arXiv:2607.09197 are source-reported preprint evidence; no local HSE or perturbation robustness reproduction was run in this pass.
An AI gateway is the control plane between an enterprise’s applications and the LLM APIs they consume. Core functions: unified API surface across multiple model providers, token-aware rate limiting, cost tracking, fallback/retry routing, RBAC, and audit logs.
The category shifted from “nice to have” to mission-critical between 2024 and 2026. Enterprises now run 5–10 model providers simultaneously across different teams. Without a central control plane, governance, cost visibility, and security become unmanageable at scale.
Market Structure
Cloud hyperscalers (bundled)
AWS Bedrock AgentCore Gateway (GA October 2025)
- Managed gateway for agent-to-agent and agent-to-tool traffic
- Built-in IAM, CloudWatch, memory, and runtime — positions as AWS’s full agentic infrastructure answer
- 100,000+ organizations on Bedrock as of mid-2026; 4.7x customer growth in the past year
- Best fit: AWS-native shops where IAM integration and CloudWatch observability are priorities
- Limitation: cloud lock-in; cross-cloud or on-prem deployments require separate tooling
Azure API Management AI Gateway (Microsoft Build 2026)
- Unified Model API translates across OpenAI, Anthropic, and Google Vertex AI behind a single surface
- Token-aware rate limiting (configurable per API key/token budget), semantic caching, content safety policies extended to MCP tool calls and agent payloads
- Reasoning tokens, cached tokens, and audio tokens tracked as separate metrics in April 2026 update
- Best fit: Azure/M365 enterprises running Copilot alongside custom LLM workloads
- Limitation: Azure-only; no routing to Bedrock-hosted Claude or other clouds
Pure-play (vendor-neutral)
LiteLLM Enterprise
- Most widely deployed open-source AI proxy; de facto standard for vendor-neutral self-hosted deployments
- 100+ provider support with unified OpenAI-format API; strong router with retry/fallback, budget controls, RBAC
- Enterprise tier adds SSO, audit logs, dedicated support
- Limitation: operational overhead of self-hosting; cached token tracking requires version discipline (see below)
Portkey (acquired by Palo Alto Networks, ~$700M, April 30 2026)
- Pre-acquisition: trillions of tokens/month, Fortune 500 customers, 3,000+ LLMs + MCP servers supported
- Strong on prompt management, versioning, evaluation, and observability
- Post-acquisition: gateway will become the AI security/routing layer inside Palo Alto’s Prisma platform
- Key risk: integration roadmap uncertainty during acquisition transition
Kong AI Gateway
- Best position: enterprises already running Kong for API management who want AI plugins on top
- AI plugins: LLM proxy, semantic caching, token-based rate limiting, Enterprise MCP Gateway (2026) for agentic traffic
- Gartner Magic Quadrant Leader in API Management for six consecutive years
- Limitation: over-engineered for greenfield AI-only deployments; strongest in existing Kong shops
Gateway-native open-source challengers
Bifrost / Maxim
- Public pricing page says OSS is free forever; Enterprise is custom pricing. A user-reported figure of $2,500/month per VPC instance was not verified on the public pricing page.
- Current public pricing/docs place virtual keys, budget/rate limits, custom routing rules, MCP Gateway, semantic caching, Prometheus metrics, Bifrost CLI, LiteLLM compatibility, JSON parser plugin, prompt repository, and Maxim integration in OSS.
- Enterprise adds guardrails, clustering, adaptive load balancing, SAML/OIDC/audit reports, vault support, log exports, in-VPC/air-gapped deployment packaging, Okta/Entra-style identity provider sync, RBAC, MCP federated auth, audit logs, external OTel connectors, commercial onboarding, and support.
- Source-tree inspection supports the split: OSS includes
plugins/governance,plugins/logging,plugins/semanticcache, and telemetry; enterprise changelogs reference separategithub.com/maximhq/bifrost-enterprise, SCIM/default-access-profile behavior, enterprise load-balanced routing, BigQuery/Datadog export dimensions, access profiles, skills repository RBAC, and cluster-governance fixes. - Fit: low-latency self-hosted gateway where coding-agent routing and virtual-key governance matter, and the buyer can live with a still-maturing enterprise/OSS boundary.
- Hard edge: if the required control is Okta/Entra user sync, custom RBAC, cluster-wide governance, immutable audit exports, or vault-backed secrets, assume Enterprise or a fork. If the required control is just per-consumer virtual keys and budget/rate limits, the public docs classify that as OSS.
FrankenGate (pierretokns/frankengate)
- Public Apache-2.0 Go project described as a Bifrost-compatible AI gateway for evidence-gated enterprise governance, virtual keys, identity, routing, MCP, and multi-pod reliability.
- Its architectural motivation is directly relevant to this report: it targets the OSS Bifrost failure boundary where database-backed virtual-key changes can remain local to one pod without a propagation mechanism.
- The repository is public, currently reports the
devbranch as its default branch, and was updated on July 19, 2026. Those metadata signals establish an active project, not production maturity. - Current evidence does not establish that FrankenGate has achieved cross-pod linearizability, atomic budget enforcement, identity synchronization, or harness compatibility. Those claims require source inspection and a runnable multi-pod test, not README intent.
- Fit: a fork/alternative to evaluate when Bifrost OSS governance is attractive but its single-pod configuration state is unacceptable. The principal question is whether FrankenGate replaces implicit in-memory state with an explicit, tested control-plane contract.
- Required proof before procurement: create/update/revoke a virtual key through pod A, immediately exercise it through pods B and C, update budgets and provider restrictions concurrently, kill the writer pod, induce a database/network partition, and verify convergence, monotonic revocation, and aggregate budget behavior.
TensorZero
- Open-source LLMOps platform rather than a pure gateway: unified provider API, observability, evaluation, optimization, and experimentation.
- Rust gateway claims <1ms p99 overhead at 10k+ QPS; supports OpenAI-compatible clients, AWS Bedrock, SageMaker, Azure, Anthropic, Vertex, OpenRouter, vLLM, SGLang, TGI, and other providers.
- Stronger than most gateways on data flywheel shape: stores inferences and feedback in the operator’s database, exposes datasets, replay, inference/workflow evals, LLM judges, SFT/optimization workflows, adaptive A/B tests, OTLP trace export, and Prometheus metrics.
- Fit: teams that want gateway control and eval/optimization feedback loops in one OSS stack.
- Hard edge: TensorZero is not primarily a virtual-key enterprise IAM product. It has auth, rate limits, credential management, relay/edge gateway patterns, tags, and cost tracking, but Bifrost/LiteLLM/Kong are closer to traditional gateway procurement checklists.
LiteLLM Rust
- Separate from the Python LiteLLM proxy. The current README positions it as a minimal Rust gateway for coding agents including Claude Code and Codex.
- It has explicit
lite claudeandlite codexlaunchers. Codex is routed through OpenAI Responses API SSE overPOST /v1/responseswithwire_api="responses"config overrides; Claude Code is launched with Anthropic-compatible environment variables. - Fit: local/coding-agent gateway experiments and latency-sensitive harness bridging.
- Hard edge: not a drop-in replacement for LiteLLM Enterprise’s mature virtual-key, spend, RBAC, admin, alerting, and enterprise ops surface.
Feature Heatmap and Latency Evidence
This is the visual comparison layer for procurement and architecture review. Yes means the capability is explicitly documented or present in the public project evidence; Partial means it exists with meaningful scope limits; No means the product is not positioned as owning that capability; Unknown means the public evidence is insufficient. The matrix is intentionally grouped by where the capability lives in the request lifecycle.
Inference hot path
| Capability | LiteLLM Python | Bifrost OSS | Bifrost Enterprise | FrankenGate | TensorZero | Kong | Cloudflare | AgentCore |
|---|---|---|---|---|---|---|---|---|
| OpenAI-compatible API | Yes | Yes | Yes | Bifrost-compatible goal | Yes | Yes | Yes | Partial |
| Anthropic-compatible API | Yes | Yes | Yes | Unknown | Partial | Partial | Partial | Partial |
| Bedrock routing | Yes | Yes | Yes | Unknown | Yes | Partial | Partial | Yes |
| Responses API / SSE | Partial | Partial | Partial | Unknown | Partial | Partial | Partial | Unknown |
| Streaming tool-call fidelity | Partial | Partial | Partial | Unknown | Partial | Partial | Partial | Partial |
| Retries and fallbacks | Yes | Yes | Yes | Goal / unverified | Yes | Yes | Yes | Partial |
| Weighted or adaptive load balancing | Yes | Partial | Yes | Goal / unverified | Yes | Partial | Partial | Partial |
| Semantic routing | Partial | Partial | Partial | Unknown | Partial | Partial | Partial | Unknown |
| Semantic caching | Partial | Yes | Yes | Unknown | Partial | Yes | Yes | Unknown |
| Model aliases and provider translation | Yes | Yes | Yes | Bifrost-compatible goal | Yes | Partial | Partial | Partial |
| Token and cost accounting on hot path | Yes | Yes | Yes | Unknown | Yes | Partial | Partial | Partial |
Governance and enterprise control plane
| Capability | LiteLLM Python | Bifrost OSS | Bifrost Enterprise | FrankenGate | TensorZero | Kong | Cloudflare | AgentCore |
|---|---|---|---|---|---|---|---|---|
| Virtual keys | Yes | Yes | Yes | Project goal / unverified | Partial | Partial | Partial | Partial |
| Per-team or per-project budgets | Yes | Yes | Yes | Unknown | Partial | Partial | Partial | Partial |
| Aggregate budget enforcement across pods | Partial | No in OSS multinode | Enterprise mechanism documented | Core proof required | Unknown | Partial | Partial | Partial |
| Per-model and provider restrictions | Yes | Yes | Yes | Project goal / unverified | Partial | Partial | Partial | Partial |
| Key rotation and secret managers | Yes | Partial | Yes | Unknown | Partial | Partial | Partial | Yes |
| SAML/OIDC SSO | Enterprise | No / limited | Yes | Unknown | Partial | Yes | Cloudflare account | IAM |
| Okta/Entra/SCIM synchronization | Enterprise | No | Yes | Identity goal / unverified | Unknown | Partial | No | IAM/IdP |
| Fine-grained RBAC | Enterprise | Partial | Yes | Governance goal / unverified | Partial | Yes | Partial | IAM/Cedar |
| Immutable audit export | Partial | Partial | Yes | Evidence goal / unverified | Partial | Yes | Partial | CloudTrail/CloudWatch |
| Policy decision versioning | Partial | Partial | Partial | Evidence goal / unverified | Partial | Partial | Partial | Partial |
LLMOps, evals, and data flywheel
| Capability | LiteLLM Python | Bifrost OSS | Bifrost Enterprise | FrankenGate | TensorZero | Kong | Cloudflare | MLflow |
|---|---|---|---|---|---|---|---|---|
| Native request/response logs | Yes | Yes | Yes | Unknown | Yes | Partial | Yes | Partial |
| OpenTelemetry traces | Yes / callback | Yes | Yes | Unknown | Yes | Partial | Partial | Partial |
| Full tool-call and tool-result replay | Partial | Partial | Partial | Evidence goal / unverified | Partial | Partial | Partial | Partial |
| Dataset construction from traces | Partial | Partial | Partial | Unknown | Yes | No | Partial | Yes |
| LLM-as-judge | Via integrations | Via integrations | Via integrations | Unknown | Yes | No | Partial | Yes |
| Workflow or trajectory evals | Via integrations | Via integrations | Via integrations | Unknown | Yes | No | No | Yes |
| Prompt/version experiment tracking | Partial | Prompt repository | Prompt repository | Unknown | Yes | Partial | Partial | Yes |
| Optimization or fine-tuning loop | No | No | No | Unknown | Yes | No | No | Partial |
| Trace-to-training provenance | Partial | Partial | Partial | Evidence goal / unverified | Partial | No | No | Partial |
| Eval promotion gate | External | External | External | Evidence goal / unverified | Partial | External | External | Partial |
Deployment, isolation, and operations
| Capability | LiteLLM Python | Bifrost OSS | Bifrost Enterprise | FrankenGate | TensorZero | Kong | Cloudflare | AgentCore |
|---|---|---|---|---|---|---|---|---|
| Self-hosted | Yes | Yes | Yes | Yes | Yes | Yes | No | No |
| VPC or private deployment | Yes | Yes | Yes | Unknown | Yes | Yes | Partial | AWS |
| Air-gapped deployment | Possible | Possible | Documented | Unknown | Possible | Possible | No | No |
| Kubernetes support | Yes | Yes | Yes | Goal / unverified | Yes | Yes | No | AWS-managed |
| Multinode live configuration sync | Partial / topology-dependent | No in OSS | Yes / clustering | Core proof required | Partial | Partial | Managed | Managed |
| Health checks and rollout safety | Yes | Yes | Yes | Unknown | Yes | Yes | Yes | Yes |
| Prometheus or metrics export | Yes | Yes | Yes | Unknown | Yes | Yes | Partial | CloudWatch |
| Regional failover | Partial | Partial | Yes | Unknown | Partial | Partial | Partial | Yes |
Reading the heatmap: Bifrost OSS has a surprisingly broad feature surface, but its multinode governance boundary is a hard negative. FrankenGate is relevant precisely because it targets that boundary; it should not receive a positive mark until the multi-pod evidence bundle exists. TensorZero leads the LLMOps/data-flywheel grouping, while LiteLLM and Bifrost lead the traditional gateway grouping.
Published latency evidence
The following is an evidence chart, not a benchmark leaderboard. Public sources do not expose a common workload, payload size, provider, concurrency, streaming definition, or percentile methodology across these products. Consequently, comparable p50/p95 end-to-end latency is NR (not reported) for all products in the current desk-research pass.
| Product | p50 end-to-end | p95 end-to-end | Public latency evidence | Evidence class |
|---|---|---|---|---|
| LiteLLM Python | NR | NR | No comparable public p50/p95 found | Unknown |
| Bifrost OSS | NR | NR | 11 us overhead at 5,000 RPS on t3.xlarge; 59 us on t3.medium; vendor benchmark, not p50/p95 | Vendor-reported overhead |
| Bifrost Enterprise | NR | NR | No separate public p50/p95 found | Unknown |
| FrankenGate | NR | NR | No public p50/p95 or reproducible latency result verified | Unknown |
| TensorZero | NR | NR | <1 ms p99 gateway overhead at 10k+ QPS; not end-to-end provider latency | Vendor-reported overhead |
| Kong AI Gateway | NR | NR | No comparable public p50/p95 found | Unknown |
| Cloudflare AI Gateway | NR | NR | No comparable public p50/p95 found | Unknown |
| AWS AgentCore Gateway | NR | NR | No comparable public p50/p95 found | Unknown |
The proper visual for the next pass is a paired p50/p95 dot plot after running identical workloads through each gateway. The required measurement record is: gateway version and commit, deployment shape, CPU/memory, provider/model/region, API path, prompt and output token counts, concurrency, warm-up policy, streaming or non-streaming mode, retry policy, sample count, timeout, and whether the value is gateway-only overhead or full provider round-trip.
Latency benchmark contract
same request corpus + same provider route + same concurrency
-> direct provider baseline
-> gateway A / gateway B / gateway C
-> p50, p95, p99 TTFT and total latency
-> error rate, retries, fallback count, token/cost drift
-> trace completeness and tool-call replay score
Do not compare Bifrost’s microsecond overhead claim or TensorZero’s p99 overhead claim directly with an end-to-end p50/p95 number. They measure different quantities. The chart should show them as separate evidence classes until the conformance runner produces a common distribution.
Adoption Signals by Segment
| Segment | Leading choice | Why |
|---|---|---|
| AWS-native enterprises | Bedrock AgentCore | IAM/CloudWatch native, no extra vendor |
| Azure/M365 shops | Azure APIM | M365 integration, Unified Model API |
| Multi-cloud / vendor-neutral | LiteLLM Enterprise | 100+ providers, self-hosted control |
| Existing Kong infra | Kong AI Gateway | Avoid dual-vendor, MCP agent traffic |
| Security-first enterprises | Portkey (post-Palo Alto) | Network security + AI governance convergence |
| Financial services | Bedrock (AWS-heavy) or APIM (MS-heavy); LiteLLM for on-prem | Regulatory + data residency requirements |
Enterprise Decision Matrix
| Product | Self-host / VPC | OSS vs enterprise boundary | Bedrock support | Routing depth | Virtual-key / identity depth | Trace/eval depth | Coding-agent harness fit | Procurement note |
|---|---|---|---|---|---|---|---|---|
| LiteLLM Python proxy | Yes | OSS has proxy, virtual keys, budgets, model routing; enterprise adds SSO, audit/retention, JWT, route control, secret managers, IP ACLs, key rotation, team logging, custom-tag budgets, model budgets for virtual keys, log export, branding | Strong, but Converse prompt-caching gap remains | Strong retry/fallback/load balancing, model groups, budgets | Strong virtual-key/team/project model; strongest if Enterprise is available | Broad callback ecosystem; content capture requires explicit privacy-sensitive toggles | Good via OpenAI-compatible APIs; Codex Responses and Claude Code behavior need route-specific tests | Most mature neutral default, but Python latency/ops and version discipline are real costs |
| Bifrost OSS | Yes | OSS publicly includes virtual keys, budgets/rate limits, routing rules, MCP gateway, semantic caching, OTel, CLI, prompt repository; multinode UI/API synchronization is explicitly unsupported | Yes | Strong custom routing/fallbacks; adaptive LB is enterprise | OSS virtual keys/teams/customers; IdP/RBAC enterprise; config changes are pod-local in multinode database-backed deployments | Built-in observability, OTel metrics/traces; external OTel connectors/log exports enterprise | Strongest published coding-agent story among gateways: Bifrost CLI names Claude Code, Codex CLI, Gemini CLI | Suitable for one node or shared-file GitOps; do not treat shared PostgreSQL as a multinode control plane |
| Bifrost Enterprise | VPC/on-prem/air-gapped | Adds clustering, adaptive LB, guardrails, SAML/OIDC, Okta/Entra-style sync, RBAC, vault, log exports, audit logs, federated MCP auth | Yes | Strongest for high-availability routing | Strongest on enterprise identity packaging | Stronger exports and audit surface | Strong, same CLI plus enterprise policy controls | Public page says custom pricing; public page did not verify $2,500/month claim |
| FrankenGate | Yes, public Apache-2.0 Go repository | Project-level fork/alternative intended to address Bifrost-compatible governance and multi-pod reliability; exact OSS/enterprise boundary is not established | Unknown pending provider-matrix verification | Intended to preserve Bifrost-compatible routing; implementation and failover behavior require source/test verification | Virtual keys and identity are stated project goals; cross-pod consistency, revocation ordering, and aggregate budgets are unverified | Evidence-gated governance is a project goal; canonical trace/export contract is unverified | Bifrost compatibility suggests a starting point, but Claude Code/Codex Responses/tool-call behavior is unverified | Evaluate as a targeted reliability fork, not as a production-proven gateway; demand a multi-pod conformance bundle |
| TensorZero | Yes | Public repo/docs present OSS gateway + observability + eval + optimization platform | Yes | Routing, retries, fallbacks, load balancing, granular timeouts | Auth, rate limits, credential management, relay/edge centralization; not classic virtual-key entitlement product | Strong: inferences, feedback, datasets, replay, evals, LLM judges, OTLP, Prometheus | OpenAI-compatible clients work; Claude Code/Codex harness auto-mode behavior not documented as a first-class feature | Best if the goal is gateway-to-eval-to-optimization flywheel, not only central API control |
| Kong AI Gateway | Yes | AI features ride Kong Gateway/Konnect packaging; enterprise value is existing API management, plugins, governance, support | Provider/plugin dependent | Strong API-policy and plugin routing; AI-specific semantic features exist | Strong where Kong already owns API identity/policy | Good infra/APM integration; LLM eval loop is not its center | Generic OpenAI-compatible routing only; harness semantics need tests | Best when Kong is already the enterprise API gateway |
| Cloudflare AI Gateway | Cloudflare-managed | Cloudflare ecosystem service | Multi-provider, but traffic enters Cloudflare control plane | Caching, rate limiting, logs, provider routing in Cloudflare model | Cloudflare account/API-token model, not per-repo VPC virtual-key system | Useful dashboard/logging; not a full eval/optimization platform | Fine for HTTP clients; less suitable for private Bedrock/VPC routing and harness-specific model alias control | Useful edge service, weak fit for regulated “all traffic stays in our VPC” requirement |
| AWS Bedrock AgentCore Gateway | AWS-managed | Managed AWS service | Native | Gateway for agent/tool traffic, IAM/Cedar/AgentCore integration | Best AWS-native IAM/Identity path | AgentCore Observability/Evaluation/CloudWatch/X-Ray native | Harness compatibility depends on client/provider adapter; not a universal OpenAI/Anthropic gateway | Best for AWS-only governance; weak for neutral multi-provider control |
| MLflow Deployments / Tracing | Self/cloud depending deployment | OSS/Databricks packaging varies | Provider adapters, not an enterprise gateway replacement | Not a full traffic gateway | Not a virtual-key gateway | Strong experiment/eval/tracing system of record | Not a harness gateway | Pair with a gateway; do not confuse it with gateway policy enforcement |
Harness Compatibility Is a Separate Gate
Most gateway comparison tables stop at “OpenAI-compatible.” That is not enough for Claude Code, Codex, Gemini CLI, or Bedrock/Nova routing through enterprise harnesses.
The hard contract is not only POST /v1/chat/completions. It is:
- Model namespace aliasing: whether the gateway can expose model names that the harness recognizes while routing to Bedrock/provider-specific targets. For Claude Code, this often means Anthropic-flavored model names and behavior. For Codex, it may mean OpenAI model IDs or configured provider aliases.
- Wire API: Codex’s modern harness path can require OpenAI Responses API streaming semantics, not only Chat Completions. LiteLLM Rust explicitly documents
POST /v1/responses; most gateway docs still emphasize Chat Completions. - Streaming event fidelity: tool calls, partial arguments, reasoning summaries, usage deltas, and final usage accounting must survive translation. A gateway that drops provider-native tool-call IDs or truncates streaming usage is not usable for eval-grade trace capture.
- Permission semantics: Claude Code auto mode and Codex permission prompts are client-side harness policies. A gateway can route and audit traffic, but it should not be assumed to preserve or enforce those permissions unless the harness is tested against real tool-call flows.
- Bedrock/Nova variance: Bedrock Converse, InvokeModel, Anthropic-compatible adapters, and Nova model payloads do not have identical fields for tool use, prompt caching, token details, and logprob-like outputs. A gateway must be tested per provider path, not per brand.
Minimum harness acceptance suite:
| Test | Why it matters |
|---|---|
| Codex Responses SSE with tool call and approval prompt | Checks whether Codex can keep its permission loop while gatewaying model traffic |
| Claude Code auto-mode prompt with tool use and denial path | Checks whether gateway aliasing preserves local policy assumptions |
| Bedrock Claude via InvokeModel and Converse | Exposes prompt caching, tool-call, and usage accounting differences |
| Bedrock Nova text/tool request | Exposes non-Anthropic Bedrock schema drift |
| Model alias collision test | Ensures claude-*, gpt-*, and internal aliases route deterministically |
| Trace replay export | Ensures prompts, tool calls, tool results, model, region, key, user/team, and costs can be reconstructed after the request |
Trace, Evals, and MLflow Boundary
Gateways and eval platforms own different contracts:
application / coding harness
-> gateway policy plane
auth, virtual keys, model aliasing, routing, fallback, provider keys, rate limits
-> provider execution plane
Bedrock, OpenAI, Anthropic, Vertex, self-hosted vLLM/SGLang
-> trace/eval plane
OTel/OpenInference/MLflow/LangSmith/TensorZero/AgentCore/CloudWatch
-> curation plane
redaction, datasets, golden sets, SFT/preference/RLVR exports
MLflow is not the gateway. It is a strong experiment, tracing, model registry, and evaluation system. The best architecture is to let the gateway enforce policy and route traffic, while MLflow or a warehouse-adjacent system stores eval datasets, traces, judge results, and promotion evidence.
TensorZero compresses the stack. TensorZero can be both gateway and trace/eval/optimization substrate. That is attractive for a greenfield learning loop, but it should be compared against MLflow as a system of record, not only against LiteLLM/Bifrost as a gateway.
Bifrost and LiteLLM need trace export discipline. They can produce gateway telemetry, logs, and callbacks, but an enterprise eval flywheel requires more than latency and cost dashboards. Preserve: raw request/response refs, provider request IDs, model alias, routed model, region, virtual key, user/team/customer, tool-call IDs, tool results, prompt/cache usage, cost, error/fallback chain, and policy decisions.
AWS X-Ray/CloudWatch vs MLflow cost model. X-Ray/CloudWatch/Application Signals are operational tracing and APM services: useful for latency, dependency maps, errors, and AWS-native audit, but priced by telemetry volume and retention/search patterns. MLflow shifts cost to storage/compute/operations and gives better eval/experiment semantics. For high-volume agent traces, do not blindly ship full prompt/tool bodies to CloudWatch. Send redacted operational spans to CloudWatch/X-Ray and preserve full sensitive evidence in a private raw vault with separate retention.
What To Standardize
Standardize the enterprise gateway interface around a canonical event model rather than a single vendor:
| Layer | Standardize | Preserve native |
|---|---|---|
| Request identity | trace_id, span_id, session_id, turn_id, user_id, team_id, virtual_key_id, policy_version |
gateway request ID, provider request ID, harness session ID |
| Model routing | model_alias, provider, provider_model, region, routing_rule_id, fallback_chain |
provider-specific model ID and route diagnostics |
| Tool calls | tool_call_id, tool name, arguments hash, result hash, status, latency |
raw harness/tool event shape |
| Cost | input/output/cache/reasoning/audio tokens, estimated cost, billable provider | provider native usage object |
| Evaluation | evaluator ID, rubric version, score, label source, golden-set ID | MLflow/TensorZero/LangSmith/AgentCore native run IDs |
| Privacy | content capture class, redaction policy, vault ref, retention class | raw encrypted prompt/response/tool body when allowed |
Do not standardize directly on gateway logs. Use gateway logs as evidence feeding a canonical trace/eval IR, then export late into MLflow datasets, TensorZero datasets, LangSmith datasets, Hugging Face datasets, SFT JSONL, DPO/KTO preference pairs, or StateBench fixtures.
Missing Coverage and Next Experiments
The remaining high-value tests are behavioral, not desk research:
- Bifrost OSS fork test: run Bifrost OSS with virtual keys, team budgets, model restrictions, MCP tool filtering, Bedrock Claude, Bedrock Nova, Codex Responses, and Claude Code. Confirm what breaks without Enterprise.
- FrankenGate multi-pod conformance test: inspect the implementation and run at least three replicas against the same backing store. Create, update, revoke, and budget a virtual key through different pods; test writer-pod death, rolling restart, stale reads, duplicate writes, partition/reconnect, and aggregate budget enforcement. Require a replayable evidence bundle before upgrading any reliability claim.
- LiteLLM vs Bifrost latency under Bedrock: measure streaming TTFT, p95/p99, and failure behavior for InvokeModel and Converse, with identical model aliases and retry rules.
- Trace completeness bakeoff: send the same tool-using agent run through LiteLLM, Bifrost, FrankenGate, TensorZero, and AgentCore. Score whether full replay is possible without provider console access.
- MLflow bridge: convert gateway traces into MLflow traces/evaluation datasets and verify that judge scores link back to the original gateway request, virtual key, model alias, and provider request ID.
- Harness permission regression: verify Claude Code auto mode and Codex approval mode across gateway aliases; fail the gateway if permissions become ambiguous or if tool-call IDs cannot be joined to traces.
- Router meaningfulness eval: run perturbation robustness and behavioral-diversity checks before adding more model providers. A large provider list is not inherently better if routing decisions are unstable or redundant.
LiteLLM Cached Token Tracking — Production Detail
AWS Bedrock
Cache token tracking is production-viable on LiteLLM as of mid-2025 fixes, but several conditions must hold:
What works:
cache_read_input_tokensandcache_creation_input_tokenstracked separately in usage object- Bedrock-specific cost calculation (PR #12488, July 2025) prices cache reads/writes at correct Bedrock rates
- Bugs fixed: double-counting in streaming (PR #25517), inflated TPM rate limiting (PR #18728), negative costs (PR #12258)
Critical gap — Bedrock Converse API: LiteLLM strips cache_control beta headers when routing through bedrock_converse to avoid API errors. Prompt caching simply does not work on the Converse path. Only the InvokeModel (legacy) path supports caching. Enterprises should confirm which Bedrock API path their integration uses.
OpenAI
- Cache reads tracked via
prompt_tokens_details.cached_tokensfrom OpenAI response - Critical gap: If
cache_read_input_token_costis not explicitly configured in model pricing, cached tokens are billed at the same rate as regular input tokens — cost tracking overstates actual spend and the 50% discount is not reflected - Azure OpenAI prompt caching broken for certain model versions (issue #18219); OpenAI-compatible providers inconsistent (issue #27191)
Operator checklist:
- Confirm InvokeModel (not Converse) for Bedrock Anthropic models
- Explicitly configure
cache_read_input_token_costandcache_creation_input_token_costin model pricing config - Run a version with all the 2025 cache fixes applied; pre-2025 versions have multiple unfixed bugs
- Test with a streaming request — streaming-specific double-counting was a separate bug
The 2026 Consolidation Signal
The Portkey/Palo Alto acquisition is the defining event of the AI gateway category cycle. The thesis: AI gateway is converging with network security, not developer experience tooling. The next wave of enterprise procurement will route AI gateway decisions through security teams, not platform engineering.
The countervailing signal: the cloud hyperscalers (AWS, Azure) are winning enterprise AI gateway share by bundling it into existing platform relationships. Enterprises with strong AWS or Azure commitments are unlikely to adopt a separate pure-play gateway — the switching cost is near zero when the vendor already owns their cloud platform.
LiteLLM remains the dominant choice wherever on-prem isolation, regulatory data residency, or vendor neutrality is a hard requirement. Financial services firms with strict data localization (EU, China, regulated US entities) are the clearest LiteLLM enterprise use case.
What Perplexity Uses
Perplexity does not use an off-the-shelf gateway. They built custom orchestration that routes queries dynamically across 20+ models — no single model handles more than 25% of queries (down from two models covering 90% of volume in January 2025). Their inference layer runs on NVIDIA Triton with disaggregated serving (separate GPU pools for prefill and decode phases). Commercial AI gateways are designed for enterprises consuming model APIs; companies serving at Perplexity’s scale (400M+ queries/month) build custom.
Router Meaningfulness Evaluation
The July 2026 Google DeepMind paper “When is Routing Meaningful?” adds a missing evaluation layer for AI gateways. Most gateway scorecards ask whether a router reduces cost or preserves task accuracy. That is not enough. A router can look good while routing across a redundant model pool, or while sending paraphrases of the same request to different actors.
Two structural checks should be added to gateway evals:
| Check | What it detects | Enterprise use |
|---|---|---|
| Behavioral diversity / HSE | Whether the routed model pool contains actors that behave differently enough for routing to matter | model-pool design, coreset selection, avoid redundant provider sprawl |
| Perturbation routing robustness | Whether meaning-preserving variants route to the same model, tool, or agent class | compliance stability, predictable caching, specialization, auditability |
This changes the gateway procurement question. Do not only ask whether the gateway supports fallback, budget routing, and provider weighting. Ask whether it can export routing decisions and replay a perturbation suite that measures:
- clean-query accuracy;
- perturbed-query accuracy;
- assignment robustness;
- model-pool diversity;
- marginal value of each additional routed model;
- failure cases where accuracy rises but robustness falls.
The practical implication is conservative: a smaller curated model pool can be better than a large vendor list if it preserves most behavioral diversity while reducing cost, policy, and audit complexity. In the paper’s source-reported experiments, fewer than ten agents recover most available diversity in EmbedLLM, and four agents do so in RouterBench. Treat those counts as benchmark-specific heuristics, not a universal cap.
Sources: LiteLLM GitHub PRs/issues, Kong 2025 Year in Review, AWS Bedrock press, Azure APIM Build 2026 release notes, Palo Alto Networks/Portkey press release, NVIDIA Perplexity technical blog, Google DeepMind arXiv:2607.09197.