← Agent Frameworks 🕐 25 min read

Agent Frameworks

AI Unit Economics for SaaS Products — LLM as Variable COGS (2026)

Executive Summary · March 2026

LLM inference has broken the foundational assumption of SaaS economics: near-zero marginal cost.

LLM inference has broken the foundational assumption of SaaS economics: near-zero marginal cost. For any product where AI is a primary value driver, inference is now a meaningful variable cost that scales with usage, not with seat count. This creates a structural mismatch between how most SaaS products are priced (flat per-seat) and how they actually incur cost (per-token, per-call, per-generation).

ICONIQ’s January 2026 benchmark reported average AI product gross margin at 52%, up from 41% in 2024. Traditional SaaS targets 75–85%. That 20–30 point gap is almost entirely explained by inference cost. The companies closing that gap — GitHub Copilot, Cursor, Notion AI — are doing so through a combination of pricing model redesign, model routing, caching, and per-feature cost attribution. This note covers each lever with working numbers.

1. The AI COGS Problem

How LLM Inference Breaks SaaS Unit Economics

Traditional SaaS has a cost structure that rewards scale. Once you build the product, serving the Nth customer costs almost nothing — a few cents of compute, a few milliseconds of database I/O. Gross margins expand as revenue grows because COGS is largely fixed: hosting, infrastructure, engineering salaries amortized across customers.

LLM inference inverts this. Every AI feature invocation burns tokens at a rate that is:

Variable by query complexity — a 10-sentence answer costs 10× more than a 1-sentence answer
Variable by model selection — Claude Sonnet at $3/$15 per million tokens versus Claude Opus 4 at $15/$75 is a 5× cost difference for the same output category
Variable by user behavior — power users who prompt-and-iterate hit 10× the token consumption of casual users on the same seat price

For a $20/month seat (like Cursor Pro), the math is brutal. If the average user runs 100 AI requests/month at an average of 2,000 output tokens each, that is 200,000 output tokens. At Sonnet pricing ($15/M output tokens in 2025), that is $3.00 in inference cost on a $20 seat — a 15% variable COGS rate before hosting, support, or amortized engineering. Shift 20% of requests to Opus ($75/M output), and the inference bill on that seat jumps to $6–$8 — approaching 40% of seat revenue.

Cursor’s public pricing crisis in summer 2025 illustrates the failure mode precisely. After switching from fixed “fast request” allotments to usage-based credits tied to actual API costs, the company issued a public apology on July 4, 2025, offering refunds when some users discovered their real costs were several times the monthly price. The June 2025 repricing happened because flat per-request pricing was becoming unsustainable as users shifted to expensive frontier models.

Gross Margin Compression Benchmarks

Company Type	Gross Margin Range (2025–2026)	Primary COGS Driver
Traditional SaaS (no AI)	78–85%	Hosting, support
AI-embedded SaaS (AI as feature)	72–80%	Inference + hosting
AI-native SaaS (AI as core product)	52–68%	Inference dominates
LLM-native (AI is the product)	41–65%	Inference + model costs

Source: ICONIQ Growth State of AI 2026 snapshot; Bessemer Venture Partners State of AI 2025; industry benchmarks aggregated from public filings and private market data.

The threshold that triggers material investor scrutiny is inference exceeding 10% of revenue. Below that, AI COGS is noise. Above 15%, it starts showing up in MD&A, analyst questions, and comp table gross margin adjustments.

Real-World Examples

GitHub Copilot: By January 2026, Copilot had 4.7 million paid subscribers at $10–$19/month (individual) or $19/user/month (enterprise). Microsoft has never disclosed per-user inference cost directly, but the June 2026 shift from flat pricing to usage-based billing (1 credit = $0.01, consumption varies by model and token count) is a direct signal that the flat model became unsustainable at scale. The credit model shifts inference cost risk back to the user — a structural acknowledgment that flat pricing cannot hold when users route to expensive frontier models.

Cursor: Offers six tiers from Hobby (free) to Ultra ($200/month). The $200/month Ultra tier launched as a response to power users hitting overage caps on the $60 Pro+ tier. A single 3-hour Claude Opus 4 coding session documented at $151 in API costs illustrates why the per-seat model only works if heavy users are priced at heavy-user rates.

Notion AI: Launched as a $10/member/month add-on (2024), then moved AI into Business tier ($20/user/month) in early 2026, effectively doubling the price for users who want AI features. The bundling strategy extracts higher per-seat revenue to cover inference without visibly raising the “AI price.”

Perplexity: $200M annualized revenue in 2025, $500M+ by April 2026. Runs at 60–75% gross margin, maintained through model routing, inference efficiency improvements, and a paid Pro tier ($20/month) that subsidizes high-cost agentic queries. The company has explicitly cited model mix management as a key gross margin lever.

Klarna: The most cited AI cost story in enterprise. Deployed an OpenAI-powered customer service assistant in February 2024; claimed $40M in annual cost avoidance at launch (2.3M chats in 30 days, equivalent to 700 FTE). By Q3 2025, the claim had grown to $60M and “853 agent equivalents.” However, Klarna reversed course in May 2025, publicly rehiring human agents as CEO Sebastian Siemiatkowski conceded cost-driven automation produces “lower quality.” The actual unit economics were more nuanced: customer service cost per transaction fell from $0.32 (Q1 2023) to $0.19 (Q1 2025) — a 40% reduction — but AI inference was a new COGS line that offset some of the labor savings.

2. Cost-Per-Feature-Use Modeling

Why “Total Inference Spend” Is the Wrong Metric

Most engineering teams start by monitoring total LLM spend. This is necessary but insufficient for product economics. The CFO question is not “how much did we spend on inference?” — it is “which features are margin-positive and which are margin-negative?”

A document summarization feature may consume 3× more tokens than a grammar correction feature but generate 5× more engagement and willingness-to-pay. Without per-feature attribution, you cannot make rational decisions about which features to subsidize, which to gate behind higher tiers, and which to re-architect.

Per-Feature Token Attribution Architecture

The architecture that makes per-feature attribution possible has three components:

1. Request tagging at the call site

Every LLM API call must carry metadata identifying the feature, not just the team or service. In practice this means:

# Bedrock via boto3 — tagging the request
response = bedrock_runtime.invoke_model(
    modelId="arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0",
    body=json.dumps({"messages": [...]}),
    # Application Inference Profile carries cost allocation tags
    # profile ARN encodes: feature=doc-summarization, tier=pro, tenant=customer-xyz
)

# OpenAI-compatible gateway (LiteLLM) — metadata in headers
import litellm
response = litellm.completion(
    model="claude-3-5-sonnet",
    messages=[...],
    metadata={
        "feature": "doc-summarization",
        "product_tier": "pro",
        "customer_id": "cust-abc123",
        "session_id": "sess-xyz"
    }
)

2. Cost allocation tag propagation

For AWS Bedrock, Application Inference Profiles (AIPs) are the native mechanism. An AIP is a tagged wrapper around a foundation model ARN — you invoke the profile ARN instead of the model ARN directly, and cost allocation tags flow automatically to Cost Explorer and CUR 2.0.

Each unique combination of feature, tier, and customer requires a separate AIP. For a multi-tenant SaaS with 5 features × 3 tiers = 15 profiles (per model). Tag key-value examples:

feature: doc-summarization, grammar-check, chat-assistant, code-completion, report-generation
product_tier: starter, pro, enterprise
customer_id: only practical at enterprise scale with dedicated infrastructure; for SMB use tier and feature tags instead

3. Query patterns for feature-level cost attribution

Once tags flow into CUR 2.0, Athena queries can break down inference spend by any dimension:

-- Monthly inference COGS by feature
SELECT
    resource_tags['feature'] AS feature,
    resource_tags['product_tier'] AS tier,
    SUM(line_item_unblended_cost) AS monthly_inference_cost,
    SUM(usage_amount) AS total_tokens
FROM cur_2_0_table
WHERE
    product_service_code = 'AmazonBedrock'
    AND line_item_usage_start_date BETWEEN DATE '2026-05-01' AND DATE '2026-05-31'
GROUP BY 1, 2
ORDER BY 3 DESC;

-- Cost-per-active-user by feature (join with product analytics)
SELECT
    b.feature,
    b.monthly_inference_cost,
    p.monthly_active_users,
    b.monthly_inference_cost / NULLIF(p.monthly_active_users, 0) AS inference_cost_per_mau
FROM bedrock_costs b
JOIN product_analytics p ON b.feature = p.feature AND b.month = p.month;

Feature-Level Economics Example

For a hypothetical $30/month B2B SaaS with three AI features:

Feature	Avg tokens/use	Uses/user/month	Monthly token cost/user	Revenue attribution
Chat assistant	4,000 output	40	$2.40 (Sonnet)	High engagement, high value
Grammar check	200 output	200	$0.60	Commodity, low cost
Report generation	8,000 output	5	$0.60	High value, low frequency
Total	—	—	$3.60	12% of $30 seat

At 12% of seat revenue, this is approaching the threshold where gross margin compression becomes material. The grammar check feature is the cheapest per-use but highest frequency — a candidate for routing to a smaller model (Haiku at $0.25/$1.25 per M tokens) without user-visible quality degradation.

3. Pricing Model Options for AI-Heavy Products

Option 1: Usage-Based Pricing (Pass-Through)

Structure: Customer pays per API call, per token consumed, or per AI feature invocation. Costs are passed through with a margin markup.

Economics: Cleanest unit economics. If you charge $0.02 per 1K output tokens and your inference cost is $0.015, you earn $0.005 margin per 1K tokens regardless of usage volume. Gross margin on the AI component is fixed.

Real examples: Anthropic’s API pricing directly. AWS Bedrock. OpenAI API. These are infrastructure-layer products where usage-based is the only rational model.

Problems for product companies:

Buyers resist unpredictable bills. Enterprise procurement wants fixed annual commitments.
Sales cycle complexity: finance teams cannot approve “up to $X/month” — they need a number.
High CAC: usage-based products require more customer education and longer trial-to-close cycles.
Churn risk: sticker shock on first high-usage invoice.

Best fit: Developer tools, API products, infrastructure platforms. Not typical SaaS.

Option 2: Tiered/Seat Pricing with AI Usage Caps

Structure: Monthly per-seat price includes a defined quantity of AI usage (e.g., “1,000 AI requests/month” or “2M tokens/month included”). Overage is charged or throttled.

Economics: Predictable revenue. Gross margin is protected if the included allotment is sized correctly. The risk is systematic under-sizing (too generous → margin compression at scale) or over-sizing (too restrictive → user frustration and churn).

Sizing the allotment: Model the P90 user, not the mean. If average usage is 500K tokens/month but P90 is 2.5M tokens, pricing to the mean means your heaviest users destroy your margin. The allotment must be set at a level where P75–P80 of users stay within it comfortably, and P90+ either self-select to a higher tier or hit overage.

Cap design options:

Hard cap (throttle): request fails when quota exhausted. Low margin risk, high churn risk.
Soft cap (overage billing): request succeeds, billed at overage rate. Unpredictable bills, user frustration.
Soft cap (queue degradation): request succeeds but is routed to a cheaper/slower model when over quota. Invisible to most users; smart for SaaS.

Real examples: Notion AI (bundled into Business tier, de facto soft cap since most users don’t exhaust it). Many B2B SaaS with AI copilot features.

Option 3: AI Add-On Pricing (GitHub Copilot Model)

Structure: Core product has no AI features. AI capabilities are a separate SKU — a named add-on at a clearly communicated incremental price.

Economics: Cleanest separation. You know exactly what customers are paying for AI, and you can price the add-on to hit a target gross margin on that component specifically. No cross-subsidization.

Pricing the add-on: The add-on price must cover:

Direct inference cost (P90 user × token rate)
Supporting infrastructure (embedding models, vector stores, retrieval pipelines)
Margin target (typically 60–70% gross on the add-on SKU itself)

If P90 inference cost per seat/month is $4.00 and you want 65% gross margin on the AI add-on, the minimum add-on price is $4.00 / (1 - 0.65) = $11.43/month. GitHub Copilot at $10–$19/month for individual developers is priced in this range.

Problems:

Add-ons reduce attach rate. Many users skip the add-on and use competing tools.
Sales friction: two-line-item deals are harder to close than one.
Competitive pressure: if competitors bundle AI at the same total price, the add-on looks like a tax.

Best fit: Mature products where the core is already well-priced and AI is genuinely distinct (not table stakes). Works when the AI capability is demonstrably superior to free alternatives.

Option 4: Consumption Credits

Structure: Customer purchases a credit bundle upfront. Credits are consumed as AI features are used. Credits do not expire (or expire slowly).

Economics: Favorable revenue recognition (credit sales recognized as deferred revenue, then recognized as consumed). Creates float. Predictable customer LTV if customers buy credits in advance. GitHub’s June 2026 switch to this model (1 credit = $0.01) is the highest-profile recent adoption.

Anthropic’s own model: Anthropic sells API credits in prepaid bundles with volume discounts. This gives Anthropic predictable cash flow while giving enterprise customers cost predictability.

Design considerations:

Credit pricing must reflect model cost differences (heavier model = more credits per call)
Customers can exhaust credits faster than expected if they shift to expensive models
Requires a credit balance UI and low-balance alerts — non-trivial product work
Unused credit liability on the balance sheet (if credits expire, there are revenue recognition questions)

Best fit: Developer-facing products, API platforms, products where usage is genuinely variable across customers.

Option 5: Hybrid — Flat Base + Variable AI Overage

Structure: Monthly seat fee covers the product plus a defined AI allotment. Usage above the allotment is billed at a per-token or per-request overage rate.

Economics: Provides floor revenue (seat fee) with margin protection on heavy users (overage). Most predictable for finance teams who can model a worst-case scenario.

Implementation: The overage rate must be set high enough to be margin-positive but low enough that customers do not feel punished. A rule of thumb: overage rate should be 2–3× the direct inference cost per unit, yielding 50–65% margin on overage revenue.

Real examples: Salesforce Einstein (usage-based AI credits above platform allotment). HubSpot AI features (tiered with heavy-user overage). Emerging standard for enterprise SaaS in 2026.

4. Target Gross Margin Math

The 75% Gross Margin Constraint

For a SaaS company targeting 75% gross margin, total COGS must not exceed 25% of revenue. COGS includes hosting, support, CS, amortized implementation costs, and AI inference.

If hosting + support + CS is already consuming 12–15% of revenue (typical for mid-market SaaS), that leaves 10–13% of revenue available for AI inference COGS before gross margin falls below 75%.

Working Examples at Different Price Points

$30/month SMB seat

Component	Monthly cost	% of revenue
Hosting/infra	$1.50	5.0%
Support/CS allocation	$2.00	6.7%
AI inference (P80 user)	$3.00	10.0%
Total COGS	$6.50	21.7%
Gross Margin	$23.50	78.3%

At this price point, $3.00/month in inference COGS is the upper limit to stay above 75% GM. That translates to roughly 200,000 output tokens/month at Sonnet pricing, or 600,000 tokens/month at Haiku pricing. Routing 70% of requests to Haiku extends the token budget by 3×.

$100/month prosumer seat

Component	Monthly cost	% of revenue
Hosting/infra	$3.00	3.0%
Support/CS allocation	$5.00	5.0%
AI inference (P80 user)	$15.00	15.0%
Total COGS	$23.00	23.0%
Gross Margin	$77.00	77.0%

At $100/month, $15 in inference COGS is sustainable. This covers approximately 1M output tokens/month at Sonnet pricing — enough for a heavy coding assistant or daily document AI use.

$500/month SMB SaaS (annual ACV: $6,000)

Component	Monthly cost	% of revenue
Hosting/infra	$10.00	2.0%
Support/CS allocation	$30.00	6.0%
Implementation amortization	$15.00	3.0%
AI inference (P80 user)	$60.00	12.0%
Total COGS	$115.00	23.0%
Gross Margin	$385.00	77.0%

$60/month in inference at $500/seat is 12% of revenue — at the edge of the comfortable zone but sustainable if model routing is implemented.

Break-Even Token Consumption Formula

For any seat price P and gross margin target GM:

Max inference COGS = P × (1 - GM) - non_inference_COGS

Max tokens (output) = Max inference COGS / (output_token_price_per_1M / 1,000,000)

Example for $50/seat, 75% GM target, $8/seat non-inference COGS:

Max inference COGS = $50 × 0.25 - $8 = $4.50/month
Max output tokens (Sonnet at $15/M) = $4.50 / ($15/1,000,000) = 300,000 tokens/month
Max output tokens (Haiku at $1.25/M) = $4.50 / ($1.25/1,000,000) = 3,600,000 tokens/month

The 12× difference between Sonnet and Haiku budgets illustrates why model routing is the highest-leverage gross margin lever available without repricing the product.

5. Bedrock-Specific Unit Economics for SaaS Products

Architecture Choice: Cross-Account vs. Shared Tenant

The fundamental architecture decision for a SaaS company using Bedrock is whether each customer gets their own AWS account (cross-account model) or all customers share a single account with logical separation.

Cross-account model (one AWS account per customer):

Native cost isolation — AWS Cost Explorer shows per-customer spend without any tagging gymnastics
Service limits are per-account, so one customer’s usage cannot starve another
IAM and network isolation are complete — strongest security/compliance posture
Overhead: account provisioning automation required; 15–20 accounts = manageable, 1,000+ = needs AWS Control Tower or similar

Shared tenant model (one AWS account, logical separation via tags):

Simpler to operate at small scale
Cost attribution requires Application Inference Profiles and diligent tagging
Service limits shared across all customers — quota contention risk at scale
Lower infrastructure management overhead

For most B2B SaaS companies, the right answer is: shared tenant below $1M ARR, cross-account above it (or when compliance requirements force it earlier for regulated industries).

Application Inference Profiles for Per-Customer Attribution

AWS introduced Application Inference Profiles (AIPs) in late 2024. The mechanism: instead of invoking a model ARN directly, you create a profile ARN that wraps the model and carries cost allocation tags. Every API call through the profile ARN emits cost records tagged with your custom metadata.

Setup pattern:

import boto3

bedrock = boto3.client('bedrock', region_name='us-east-1')

# Create one profile per customer (or per feature × tier combination)
profile = bedrock.create_inference_profile(
    inferenceProfileName='customer-acme-corp-sonnet',
    description='ACME Corp - Claude Sonnet 3.5 v2',
    modelSource={'copyFrom': 'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0'},
    tags=[
        {'key': 'customer_id', 'value': 'acme-corp'},
        {'key': 'customer_tier', 'value': 'enterprise'},
    ]
)

# Invoke through the profile ARN — costs automatically tagged
response = bedrock_runtime.invoke_model(
    modelId=profile['inferenceProfileArn'],
    body=json.dumps({"messages": [...]})
)

Tags flow to CUR 2.0 within 24 hours. Cost Explorer can then show a customer_id dimension. This is the operational foundation for customer-level cost-to-serve tracking.

Limitation: You need a separate profile for every unique tag combination × model. For a 3-model × 3-tier setup serving 500 enterprise customers, that is 9 profile types, but if you tag per-customer that is 500 × 9 = 4,500 profiles. AWS recommends tagging at the tier level and using CloudWatch custom metrics for customer-level granularity if profile count becomes unwieldy.

Batch Inference for Background Jobs

AWS Bedrock batch inference runs asynchronous jobs at 50% of on-demand pricing. Results are delivered to S3. This is not suitable for interactive features but is a significant lever for background workloads.

Workloads that qualify:

Document ingestion pipelines (embed and chunk new content)
Nightly report generation
Bulk classification or tagging jobs
Retroactive processing (e.g., categorize past support tickets)
Embedding generation for new content before it goes live

Economics: If your background inference is 30% of total token volume, routing it to batch cuts that 30% by 50%, reducing total inference spend by 15%. At $10,000/month total inference, that is $1,500/month without any code changes to the user-facing product.

Operational requirements: Jobs must tolerate up to 24-hour completion time. Output goes to S3; your application must poll for completion or use SNS notifications. Max batch job size: 50,000 records.

Prompt Caching for System Prompt Amortization

Most production AI features use a substantial system prompt — instructions, persona, tool definitions, context documents. If the system prompt is the same across 90% of requests (which is typical for product AI features), prompt caching eliminates the cost of re-tokenizing it on every call.

Bedrock and Anthropic both support prompt caching. Cache writes cost 25% more than standard input pricing; cache reads cost 10% of standard input pricing. For a 4,000-token system prompt:

Without caching: 4,000 tokens × $3/M = $0.012 per request
With caching (cache hit): 4,000 tokens × $0.30/M = $0.0012 per request (after first write)
Break-even: caching pays off after the 1st cache read (write cost is 1.25× standard, read is 0.1× standard → net saving on any request after the first)

For a product feature with 100K requests/day using a 4,000-token system prompt and a 90% cache hit rate:

Without caching: 100K × 4K tokens × $3/M = $1,200/day
With caching:    10K × 4K × $3.75/M (write) + 90K × 4K × $0.30/M (read) = $150 + $108 = $258/day
Saving: $942/day = $28,260/month

This is one of the highest-ROI optimizations available for production AI products.

6. Controlling AI COGS as Scale Grows

Model Routing: The Primary Cost Lever

The single highest-impact optimization for AI COGS at scale is routing requests to the cheapest model that can handle them. Not every request needs Opus. Most do not need Sonnet.

Complexity tiers and model mapping (2026 pricing):

Complexity	Model	Input $/M	Output $/M	Use cases
Low	AWS Nova Micro	$0.035	$0.14	Intent classification, slot extraction, yes/no decisions, routing
Low	Claude Haiku 3.5	$0.80	$4.00	Short summaries, form validation, keyword extraction
Medium	Claude Sonnet 3.5	$3.00	$15.00	General chat, document analysis, code assistance
High	Claude Opus 4	$15.00	$75.00	Complex reasoning, multi-step planning, research synthesis

Blended cost with routing (example distribution: 40% Micro, 35% Haiku, 20% Sonnet, 5% Opus):

Blended output cost = 0.40 × $0.14 + 0.35 × $4.00 + 0.20 × $15.00 + 0.05 × $75.00
                    = $0.056 + $1.40 + $3.00 + $3.75
                    = $8.206 / M output tokens

vs. Sonnet-only: $15.00 / M output tokens
vs. Opus-only:   $75.00 / M output tokens

The routed blend costs 45% less than Sonnet-only and 89% less than Opus-only. Teams that implement tuned routing report bill reductions of 40–85% without user-visible quality degradation, according to LiteLLM production benchmarks.

LiteLLM as the routing layer: LiteLLM provides an OpenAI-compatible API gateway with built-in cost-based routing, 100+ provider support, per-user budgets, and virtual keys. Configuration example:

# litellm_config.yaml
model_list:
  - model_name: "ai-low"
    litellm_params:
      model: "bedrock/amazon.nova-micro-v1:0"
      
  - model_name: "ai-medium"
    litellm_params:
      model: "bedrock/anthropic.claude-3-5-haiku-20241022-v1:0"
      
  - model_name: "ai-high"
    litellm_params:
      model: "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"

router_settings:
  routing_strategy: "cost-based-routing"
  
general_settings:
  max_budget: 10000  # $10K/month total ceiling

Your application code classifies request complexity (a lightweight classifier itself costing <$0.001 per call) and calls the appropriate model name.

Caching Strategies Beyond Prompt Caching

Semantic caching: If two users ask substantially the same question (“What are our refund policies?”), return a cached response rather than re-inferencing. Tools like LiteLLM’s built-in semantic cache or Redis with vector similarity check whether incoming requests are semantically equivalent to recent queries above a similarity threshold.

Typical cache hit rate for FAQ-style queries: 20–40%
Zero inference cost on a cache hit (only embedding cost to check the cache: ~$0.0001)
TTL must match content freshness requirements

Response caching for deterministic outputs: Some AI tasks are effectively deterministic for a given input — sentiment classification on the same text, entity extraction from the same document. Simple KV caching with a hash of the input achieves 100% cost elimination on repeated identical inputs.

Fine-Tuning to Replace General Models

A fine-tuned smaller model can match a large general model on specific tasks at 5–20× lower inference cost. The economics work when:

The task is narrow and well-defined (e.g., classify support tickets into 12 categories)
Training data volume is sufficient (typically 1,000–10,000 labeled examples minimum)
The task volume is high enough to amortize training cost

Rule of thumb: Fine-tuning pays off when the monthly inference cost on the target task exceeds the training cost within 3 months.

Example: Fine-tuning Claude Haiku on a support ticket classifier costs approximately $5,000–$15,000 in training cost (including data preparation). If the current cost of running Sonnet on this task is $8,000/month and a fine-tuned Haiku reduces it to $1,500/month, payback period is 2–3 months.

AWS Bedrock supports fine-tuning for Amazon Nova models and select third-party models. Custom model inference runs through the standard Bedrock API with the same AIP tagging for cost attribution.

Self-Hosting Threshold

Self-hosting LLMs on dedicated GPU infrastructure makes economic sense when the combination of token volume, data residency requirements, and latency SLA cannot be met by managed API services.

2026 break-even benchmarks:

Against frontier models (Claude Sonnet, GPT-4o): break-even at 100–256M tokens/month, depending on GPU utilization rate
Against budget APIs (Nova Micro, DeepSeek V3): self-hosting rarely beats managed pricing due to low managed rates
The critical caveat: GPU utilization must exceed 60–70% to make self-hosting competitive. At 10% utilization, effective cost per token is 10× the nominal GPU cost, which is more expensive than premium APIs.

True cost of self-hosting (one A100 80GB, on-demand AWS):

GPU instance: ~$3.20/hour = $2,304/month
Serving capacity: ~50M tokens/month at 70% utilization on Llama-3 70B or equivalent
Effective cost: ~$46/M tokens (for a 70B model)
Comparable API cost: Claude Haiku at ~$4/M output tokens

Self-hosting Llama-class models to replace Haiku does not break even on cost alone until well above 100M tokens/month and only if utilization stays high. The real drivers for self-hosting are data residency, regulatory requirements, latency, and customization — not cost savings at moderate scale.

Reserved GPU instances reduce the hourly rate by 40–60%, improving the break-even math materially for teams committed to a specific scale.

7. Investor and Board Reporting for AI COGS

How to Present on the Income Statement

There is no GAAP standard specifically for AI inference costs as of 2026. Companies have three presentation approaches:

Option A: Inference in Cost of Revenue (most common) AI inference is classified as a direct cost of delivering the product — similar to hosting costs — and appears in Cost of Revenue. This is correct and aligns with how cloud compute has always been treated.

Income statement presentation:

Revenue:                    $10,000,000
Cost of Revenue:
  Hosting & infrastructure     800,000
  AI inference costs         1,800,000   ← explicit line item
  Support & CS                 600,000
  ─────────────────────────────────────
Total COGS:                  3,200,000
Gross Profit:                6,800,000   (68% gross margin)

Option B: Inference in Cost of Revenue with MD&A disclosure Same income statement treatment, but the MD&A section of earnings reports includes explicit commentary on inference cost trends, cost per unit, and trajectory. This is the emerging standard for public AI companies in 2026 after several earnings calls where analysts asked pointed questions.

MD&A language pattern: “AI inference costs increased to $X in Q1 2026, representing Y% of revenue, compared to Z% in the prior quarter. The increase reflects [growth in AI feature adoption / shift to heavier models / expansion into new AI workflows]. We expect [trend directional commentary].”

Option C: Separate “AI-Adjusted” presentation (emerging) Some investors and analysts are building “AI-adjusted gross margin” into their models — gross margin before AI inference costs — to separate the underlying software economics from the inference layer. This is analogous to how SaaS companies disclose “non-GAAP gross margin” excluding stock compensation.

The argument for this presentation: AI inference costs are expected to decline over time as model efficiency improves (costs fell 78% through 2025 for comparable capability), so the “structural” gross margin of the software is higher than the current reported GM. Companies that present this metric argue it shows investors what margins will look like as inference costs normalize.

The counterargument: investors have seen “adjusted” metrics abused too often. Presenting AI-adjusted GM is acceptable in board decks and investor relations communications; using it as the primary reported metric in public filings draws scrutiny.

What Investors Ask About AI Gross Margins

Based on public earnings transcripts from Q4 2025 and Q1 2026, the recurring analyst questions are:

“What percentage of revenue is AI inference cost?” — They want this as a reported metric, not buried in aggregate COGS. Companies that disclose explicitly earn analyst credit; companies that bundle inference into “infrastructure” draw skeptical follow-ups.
“How does inference cost scale with revenue?” — Is AI COGS growing faster or slower than revenue? Sub-linear scaling (efficiency gains outpacing growth) is the bullish story. Super-linear scaling (more users = disproportionately more expensive users) is a red flag.
“What is your gross margin trajectory as AI usage deepens?” — If the product’s AI features become more embedded over time, does that help or hurt margins? The answer should be “helps, because we’re routing smarter and caching more.”
“How do you think about model cost risk?” — What happens to your P&L if Claude or GPT pricing increases 50%? Do you have multi-provider optionality? Is your pricing model structured to absorb cost increases?
“What is your cost per inference call, and how is it trending?” — Board-level metric. Should be tracked monthly alongside MAU and NRR.

Board-Level AI COGS Dashboard

Minimum set of metrics for a board package:

Metric	Frequency	Target
Total inference spend	Monthly	Track trend; benchmark against peer cohort
Inference as % of revenue	Monthly	Below 15% for embedded AI; below 25% for AI-native
Inference cost per MAU	Monthly	Declining QoQ as routing/caching matures
Gross margin (reported)	Quarterly	≥75% for embedded AI SaaS
Cache hit rate	Monthly	>40% for high-volume features
Top 3 features by inference spend	Monthly	Informs pricing and tier design decisions
P90 user inference cost vs. seat price	Quarterly	P90 cost must be below seat price × (1 - GM target - non-AI COGS %)

8. Real-World Benchmarks

AI COGS as Percentage of Revenue — Industry Data

Company / Cohort	AI COGS % of Revenue	Source
ICONIQ AI portfolio average (2024)	59% total COGS → ~41% GM	ICONIQ State of AI Jan 2026
ICONIQ AI portfolio average (2025)	~55% total COGS → ~45% GM	ICONIQ State of AI Jan 2026
ICONIQ AI portfolio average (Q1 2026)	~48% total COGS → ~52% GM	ICONIQ State of AI Jan 2026
Bessemer AI-native cohort (2025)	~65% GM	Bessemer State of AI 2025
AI-embedded SaaS (AI as feature)	~72–80% GM	Industry benchmarks
Inference alone at scaling-stage AI B2B	~23% of revenue	Aggregated industry estimate
Traditional SaaS (no AI)	~78–85% GM	Standard benchmarks

The trajectory is upward — from 41% GM in 2024 to 52% GM in early 2026 — driven by inference cost deflation (costs fell ~78% for comparable capability through 2025) and maturing cost optimization practices.

GitHub Copilot

4.7M paid subscribers as of January 2026; ~$700–800M ARR estimated
Pricing: $10/month individual, $19/user/month enterprise; Business tier $19/user/month
June 2026: switched to usage-based billing (1 credit = $0.01) for AI calls beyond base allotment
The pricing shift is a public admission that the flat-fee model is unsustainable as users shift to expensive frontier models (GPT-4o, Claude Sonnet) within Copilot
Implication: Microsoft’s internal unit economics on Copilot were under pressure from heavy users routing to expensive models

Cursor

Six tiers from Hobby (free) to Ultra ($200/month)
July 2025 pricing crisis: users on Pro ($20/month) found real usage costs several times higher than the monthly price when using Claude Opus 4; company issued refunds
Root cause: 1 Opus session (3 hours) = $151 in API costs vs. $20/month seat price — economics only work if heavy model users are on the $60+ tiers
Resolution: added explicit model-tier labeling in credit pools; redesigned credit system to make model costs visible to users before consumption
Lesson: opacity around model costs creates both user trust and unit economics problems simultaneously

Perplexity

$500M+ annualized revenue as of April 2026 (up 335% YoY)
Gross margins 60–75% — maintained through model routing and inference efficiency
$20/month Pro tier subsidizes heavy agentic use; free tier runs on cheaper model mix
Key lever cited in investor materials: “model mix management” — not routing users to frontier models by default

Klarna

Customer service cost per transaction: $0.32 (Q1 2023) → $0.19 (Q1 2025), a 40% reduction
AI assistant handling 67% of chats in first month (February 2024), 2.3M conversations
Reported $40M annual cost avoidance at launch; grew to $60M by Q3 2025
But: reversed course May 2025, rehired human agents for complex cases — CSAT dropped on 5% of conversations, compliance concerns on AI-only dispute resolution
Net lesson: AI COGS savings are real and material, but the “replace everything” narrative overstates the economics. The right model is AI handling high-volume simple cases, humans handling complex/emotional cases — which changes the inference cost math.

Notion AI

Launched as $10/seat/month add-on (2024), moved to Business tier ($20/seat/month) bundle in early 2026
Effective price increase: users who want AI features must pay $20/seat instead of base plan + $10 add-on
Net revenue uplift: significant (forced upgrade to Business tier), with AI inference cost covered by higher ARPU
Pricing strategy takeaway: bundling AI into higher tiers extracts more total revenue without a visible “AI price increase”

9. Operational Playbook Summary

For CTOs: What to Instrument Now

Tag every LLM call at the feature level, not just the team level. This is the prerequisite for all subsequent optimization decisions.
Deploy LiteLLM or a comparable gateway as the routing and observability layer. Avoid calling provider APIs directly from application code — you lose cost visibility and routing flexibility.
Enable prompt caching on any feature with a system prompt > 1,000 tokens. This is the fastest ROI optimization available (90%+ cost reduction on system prompt tokens at >50% cache hit rate).
Identify your P90 user’s inference cost for each AI feature. If P90 cost exceeds 20% of seat price for that feature, you have a margin problem that repricing or routing must solve.
Set up batch inference for all background workloads (document ingestion, report generation, embedding pipelines). This is a free 50% cost reduction on those workloads.

For CFOs: What to Model

Inference cost as % of revenue is your primary AI COGS metric. Track it monthly. 15% is the soft warning threshold; 25% requires structural action.
Build a per-feature P&L using cost attribution tags. Some features will be margin-positive at current usage rates; some will not. You need this to make intelligent pricing and packaging decisions.
Model inference cost deflation into your projections. If comparable inference capability cost 78% less in 2025 than 2024, assume 30–50% further deflation over the next 18–24 months. Your AI COGS as % of revenue should improve even at flat pricing.
Have an answer for investors on inference cost risk — specifically, what happens if major providers (Anthropic, OpenAI) raise prices. The answer should involve multi-provider optionality and model routing flexibility.

For AI Platform Leaders: Margin Optimization Priority Stack

In rough order of ROI-to-effort ratio:

Prompt caching — hours of engineering work; 30–90% reduction on cached tokens
Batch inference routing — days of work; 50% reduction on qualifying background workloads
Model routing (easy tasks to cheap models) — 1–2 weeks; 40–85% blended cost reduction
Semantic response caching — 1–2 weeks; 20–40% hit rate on FAQ-type queries
Fine-tuning for high-volume narrow tasks — 4–8 weeks; 5–20× cost reduction on specific tasks
Self-hosting — months; only justified by compliance requirements or >100M tokens/month at sustained utilization

The first three items alone can typically reduce AI COGS by 50–70% relative to a naive “call Sonnet for everything” baseline.