← Knowledge Base 🕐 5 min read
Knowledge Base

AI Gateway

· March 2026

The control plane between enterprise applications and LLM APIs — providing unified routing, token-aware rate limiting, cost attribution, RBAC, and audit logs across multiple model providers.

An AI gateway is the control plane that sits between an enterprise’s applications and the LLM APIs they consume. Core functions: unified API surface across multiple model providers, token-aware rate limiting, cost tracking per team or use-case, fallback and retry routing, role-based access control, and audit logs.

The category moved from “optional infrastructure” to production requirement between 2024 and 2026. Enterprises running 5–10 model providers simultaneously across different teams cannot manage governance, cost visibility, or security without a central control plane. The question is no longer whether to have one, but which architecture fits the enterprise’s existing platform.

Why this matters to mid-market buyers

  • Without a gateway, token spend is invisible by team. A single LLM API key shared across teams produces one aggregate spend number. A gateway with virtual keys and spend logging breaks that number into per-team, per-use-case, and per-workflow allocations — the prerequisite for any AI FinOps practice.
  • Model fallback and retry logic belongs in infrastructure, not application code. When a provider rate-limits or goes down, application code that handles retries inline creates N slightly different failure-handling patterns. A gateway centralizes this — one fallback policy, one audit trail, one place to adjust.
  • Gateway choice is now partly a security decision. The April 2026 acquisition of Portkey by Palo Alto Networks (~$700M) signals the market thesis: AI gateway is converging with network security, and the next wave of enterprise procurement will route these decisions through security teams, not platform engineering.

Market Structure (mid-2026)

Cloud-bundled gateways

AWS Bedrock AgentCore Gateway (GA October 2025) — Managed gateway for agent-to-agent and agent-to-tool traffic built on Bedrock. IAM and CloudWatch native; positions as the full agentic infrastructure answer for AWS shops. 100,000+ organizations on Bedrock as of mid-2026 (4.7x customer growth in the past year). Limitation: cloud lock-in; cross-cloud deployments require separate tooling.

Azure API Management AI Gateway (Microsoft Build 2026) — Unified Model API translates across OpenAI, Anthropic, and Google Vertex AI behind a single surface. Token-aware rate limiting per API key, semantic caching, content safety policies extended to MCP tool calls. Reasoning tokens, cached tokens, and audio tokens tracked as separate metrics (April 2026 update). Best fit: Azure/M365 enterprises running Copilot alongside custom LLM workloads.

Vendor-neutral gateways

LiteLLM Enterprise — Most widely deployed open-source AI proxy; de facto standard for vendor-neutral self-hosted deployments. 100+ provider support with unified OpenAI-format API; strong router with retry/fallback, budget controls per virtual key, and RBAC. Enterprise tier adds SSO (Okta, Azure AD, OIDC/SAML) and dedicated support. Proxy overhead is ~3.25ms per call at 4,200+ RPM on a 4-core instance. Limitation: operational overhead of self-hosting; PostgreSQL and Redis required for HA; several known Kubernetes deployment failure modes require workarounds.

Portkey (acquired by Palo Alto Networks, ~$700M, April 30 2026) — Pre-acquisition: trillions of tokens/month, Fortune 500 customers, 3,000+ LLMs and MCP servers supported. Strong on prompt management, versioning, evaluation, and observability. Post-acquisition: gateway will become the AI security/routing layer inside Palo Alto’s Prisma platform. Current risk: integration roadmap uncertainty during transition.

Kong AI Gateway — Best fit for enterprises already running Kong for API management. AI plugins add LLM proxy, semantic caching, token-based rate limiting, and Enterprise MCP Gateway (2026) for agentic traffic. Gartner Magic Quadrant Leader in API Management for six consecutive years. Over-engineered for greenfield AI-only deployments.

Adoption Signals by Segment

Segment Leading choice Primary reason
AWS-native enterprises Bedrock AgentCore IAM/CloudWatch native, no additional vendor
Azure/M365 shops Azure APIM M365 integration, Unified Model API
Multi-cloud or vendor-neutral LiteLLM Enterprise 100+ providers, self-hosted data control
Existing Kong infrastructure Kong AI Gateway Avoid dual-vendor, MCP agent traffic
Security-first enterprises Portkey (post-Palo Alto) Network security plus AI governance in one platform
Financial services (strict data residency) LiteLLM for on-prem; Bedrock or APIM if AWS/Azure-native Regulatory and data localization requirements

Key Operational Considerations

LiteLLM cached token tracking requires version discipline. Bugs in pre-2025 versions caused double-counting in streaming, inflated TPM rate limiting, and negative cost calculations. The Bedrock Converse API path strips cache_control headers — prompt caching only works via the InvokeModel (legacy) path. Operators should confirm which path their integration uses and explicitly configure cache_read_input_token_cost in model pricing config.

Cost attribution through a gateway requires tagging strategy. When all traffic routes through a single IAM role (LiteLLM on AWS), AWS CUR shows one aggregate line_item_iam_principal. Per-team attribution requires either STS AssumeRole per tenant with session tags (flows to CUR 2.0 iamPrincipal/ columns) or LiteLLM’s internal SpendLogs with virtual keys per team (internal chargeback, not invoice-accurate).

Gateway consolidation thesis vs. hyperscaler bundling: The Portkey/Palo Alto acquisition points toward gateway-as-security-layer. The countervailing signal is that AWS and Azure are winning enterprise gateway share by bundling into existing platform commitments. Enterprises with strong cloud commitments are unlikely to adopt a separate pure-play gateway — the marginal switching cost approaches zero when the platform vendor already owns the cloud.

Practitioner voices (pillar 13)

No named F500 executive quotes available in pillar 13 corpus on this specific topic as of June 2026. See supporting research for technical evidence.

Supporting research

See also