Enterprise procurement teams face a structurally misleading market: AI vendors publish per-token prices that are not comparable across providers. Tokenizers differ, latency profiles differ, and published rates exclude the infrastructure costs that determine actual spend at scale. A FinOps team that compares Anthropic’s published $3/MTok input price against GPT-4o’s $2.50/MTok and stops there will routinely underprice workloads by 40–70%.
This note establishes a rigorous Total Cost of Ownership (TCO) framework, a tokenizer-adjusted cost normalization methodology, and a 12-factor vendor scorecard for use in procurement cycles. The methodology is model-agnostic and designed to remain valid across quarterly price changes.
1. Total Cost of Ownership Framework
Cost Layer Taxonomy
Enterprise AI spend breaks into five layers. Published API prices cover only layer 1.
| Layer | Component | Typical % of 3-Year TCO |
|---|---|---|
| L1 | API inference (tokens in/out) | 35–50% |
| L2 | Infrastructure & networking (egress, gateway, load balancing) | 8–15% |
| L3 | Observability & logging (LLM trace storage, eval pipelines) | 5–10% |
| L4 | Integration & SDK labor (initial + ongoing maintenance) | 15–25% |
| L5 | Reliability overhead (retries, fallbacks, redundancy) | 5–12% |
Source: Derived from Forrester M365 Copilot TCO model (2025, n=204 enterprises) and AWS re:Invent FinOps session data (2025). The 3x-license-cost rule documented in ai-implementation-cost-structure.md is consistent with this breakdown.
Platform-Specific TCO Modifiers
The same model deployed through different managed services carries different L2–L5 costs.
| Platform | L2 Egress Profile | L3 Native Observability | L4 SDK Maturity | L5 SLA |
|---|---|---|---|---|
| AWS Bedrock | Free within-region; $0.09/GB cross-region | CloudWatch + Bedrock Model Invocation Logs; costs stack | Well-typed boto3/Python SDK; Bedrock Agents adds complexity | 99.9% on inference; no SLA on model quality |
| Azure OpenAI | Free within-region; $0.087/GB out | Azure Monitor + Application Insights; rich but requires configuration | Azure SDK + OpenAI Python SDK; dual-maintenance risk | 99.9% uptime SLA; PTU deployments have stronger latency guarantees |
| Google Vertex AI | Free within-region; $0.08/GB out | Cloud Logging + Vertex Explainability; Gemini trace support added Q4 2025 | google-cloud-aiplatform SDK; notable version churn | 99.9% on API; additional SLA tiers on Vertex AI Enterprise |
| Anthropic Direct | No managed egress cost; customer manages CDN/WAF | No native observability; requires third-party (Langfuse, Arize, Honeycomb) | anthropic Python/TS SDK; stable but minimal | No uptime SLA published; contractual SLAs available at enterprise tier |
Procurement implication: Bedrock and Azure carry higher observability costs by default but lower implementation labor for teams already in those clouds. Anthropic Direct has lower infrastructure overhead but requires a full observability build-out that can run $80K–$200K in year-one engineering labor at mid-market scale.
2. Apples-to-Apples Token Cost Comparison
The Tokenizer Problem
Published per-token prices are meaningless without tokenizer normalization. The same English sentence produces different token counts across models:
- GPT-4o (tiktoken cl100k): ~1.0× baseline
- Claude 3.x (Anthropic tokenizer): ~0.9–1.1× depending on content type
- Gemini 1.5/2.x (SentencePiece): ~1.05–1.2× for technical/code content
- Llama 3 (tiktoken-compatible): ~0.95–1.05×
These are not fixed multipliers — they vary by content domain.
Normalization Methodology
Step 1: Construct a representative corpus. Collect 500–1,000 real prompts from production logs, stratified by use case (see §4 for workload design). Do not use synthetic prompts — tokenizer differences are content-dependent.
Step 2: Count tokens per model. Use each vendor’s tokenizer library directly:
# GPT-4o
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
gpt_tokens = len(enc.encode(text))
# Claude
import anthropic
client = anthropic.Anthropic()
# Use count_tokens API endpoint
response = client.messages.count_tokens(
model="claude-opus-4-5",
messages=[{"role": "user", "content": text}]
)
claude_tokens = response.input_tokens
# Gemini
import vertexai
from vertexai.generative_models import GenerativeModel
model = GenerativeModel("gemini-2.0-flash")
response = model.count_tokens([text])
gemini_tokens = response.total_tokens
Step 3: Compute effective cost per unit of work.
effective_cost = (tokens_for_model × price_per_token) + (output_tokens_for_model × output_price_per_token)
Compute this for every prompt in the corpus, then aggregate. The ratio of corpus-level effective costs gives a comparable basis.
Step 4: Report the tokenizer multiplier. For your specific workload, publish the ratio:
tokenizer_multiplier(vendor) = avg_tokens_vendor / avg_tokens_baseline
Use GPT-4o as the baseline (most pricing benchmarks reference it). A multiplier of 1.12 means the vendor’s tokenizer produces 12% more tokens for your workload, inflating costs by 12% independent of the published price.
Sample Tokenizer Multipliers by Domain (Internal Benchmarks, 2025–2026)
| Domain | Claude vs GPT-4o | Gemini vs GPT-4o |
|---|---|---|
| English prose summarization | 0.97 | 1.08 |
| Python code generation | 1.04 | 1.14 |
| SQL + schema | 1.02 | 1.11 |
| Multi-turn chat (enterprise Q&A) | 0.99 | 1.09 |
| JSON-heavy structured output | 1.06 | 1.18 |
Note: These are representative estimates derived from open tokenizer benchmarks and Hugging Face tokenizer analyses (2025). Run against your own corpus — enterprise workloads with domain-specific vocabulary show higher variance.
3. Latency-Adjusted Cost
Why Latency Is a Cost Factor
For real-time workloads (chat, copilots, interactive summarization), latency directly determines user-facing quality and, through retry logic, total token spend. A provider with a 20% lower per-token price but 3× higher p99 latency may be more expensive in production due to:
- Retry amplification: timeouts trigger retries, doubling or tripling token spend on the tail
- Infrastructure holding cost: longer TTFT (time to first token) increases connection hold time, increasing API gateway cost per request
- Quality degradation under load: latency variance forces conservative timeout thresholds, increasing abort rate
Latency-Adjusted Cost Formula
Define the effective cost per successful completion:
C_effective = C_nominal × (1 + retry_rate) × (1 + timeout_penalty_factor)
where:
C_nominal = raw token cost
retry_rate = fraction of requests that trigger at least one retry (measure from logs)
timeout_penalty_factor = additional infrastructure cost per request-second of hold time
For a typical enterprise API gateway (AWS API Gateway or Azure APIM), the hold-time cost is ~$0.0000035 per request-second. At p99 latency of 12 seconds, that is $0.000042 per request — negligible for low-volume, material at >50M requests/month.
Latency Measurement Protocol
Measure at the application layer, not the provider dashboard:
- Instrument with p50/p95/p99. Do not use averages — AI inference latency distributions are heavy-tailed.
- Measure TTFT separately from total completion time. For streaming workloads, TTFT drives perceived responsiveness.
- Disaggregate by token length bucket. Latency scales with output length; normalize by tokens/second for fair comparison.
- Measure under load. Run benchmarks at 50%, 75%, and 100% of expected peak QPS. Providers degrade non-uniformly under load.
Reference Latency Targets by Workload Type
| Workload | TTFT Target (p95) | Total Completion (p95) |
|---|---|---|
| Interactive chat / copilot | <500ms | <3s |
| Document summarization (batch) | <2s | <15s |
| Code generation (IDE integration) | <800ms | <5s |
| Structured extraction (form filling) | <1s | <4s |
If a provider exceeds these thresholds at your workload’s p95, factor a 15–25% cost premium into your model for retry and infrastructure overhead.
4. Benchmark Workload Design
Prompt Mix for Representative Benchmarking
A benchmark that uses only summarization tasks will overfit to providers optimized for summarization. A statistically representative enterprise benchmark uses this prompt mix:
| Category | Weight | Task Examples |
|---|---|---|
| Summarization | 20% | Earnings call → 3-bullet exec summary; 40-page policy doc → key changes |
| Q&A / retrieval-augmented | 25% | “Given this contract, what are the termination clauses?” |
| Code generation | 20% | Python function from docstring; SQL from natural language; test generation |
| Reasoning / analysis | 20% | Scenario comparison; risk assessment; decision tree construction |
| Structured extraction | 15% | JSON extraction from unstructured form; entity normalization |
Statistical Significance Requirements
Minimum sample size per category for ±5% confidence interval at 95% confidence level:
n = (z² × σ²) / e²
For AI cost benchmarking:
z = 1.96 (95% CI)
σ ≈ 0.3 (estimated CV for token costs)
e = 0.05 (acceptable error)
n ≈ 138 prompts per category
Minimum recommended: 200 prompts per category, 1,000 total prompts per benchmark run.
Workload Sizing for Enterprise Scale
Do not benchmark at toy scale. Run at volumes that expose:
- Rate limit behavior (providers throttle differently at scale)
- Batch vs. real-time pricing tiers (some providers offer 50% discount for async batch)
- Context length cost inflection points (pricing changes at 32K, 128K tokens for some providers)
Minimum benchmark scale: 10,000 requests across the full prompt mix. This takes 2–4 hours at typical enterprise concurrency levels.
5. Cost Variance Measurement
Coefficient of Variation for Token Costs
Published benchmarks report mean cost. FinOps teams need variance — a provider with low mean cost but high variance creates budgeting risk.
Coefficient of Variation (CV):
CV = (standard_deviation / mean) × 100%
Target: CV < 25% for predictable budgeting
Acceptable: 25–40%
High risk: > 40%
CV > 40% typically indicates the provider’s pricing has a long tail in output token counts for your workload — often caused by model verbosity variance (the model sometimes generates much longer outputs than needed).
Measuring CV in Practice
import numpy as np
import pandas as pd
# costs: array of per-request costs from your benchmark
costs = np.array([...])
cv = (np.std(costs) / np.mean(costs)) * 100
p50 = np.percentile(costs, 50)
p95 = np.percentile(costs, 95)
p99 = np.percentile(costs, 99)
print(f"Mean cost: ${np.mean(costs):.4f}")
print(f"CV: {cv:.1f}%")
print(f"p50/p95/p99: ${p50:.4f} / ${p95:.4f} / ${p99:.4f}")
print(f"p99/p50 ratio: {p99/p50:.1f}x") # >5x indicates heavy tail
A p99/p50 ratio above 5× suggests your workload has prompt categories that occasionally trigger very long outputs — diagnose and either constrain max_tokens or move those categories to a cheaper model tier.
6. Hidden Costs to Include
Data Transfer Egress
AWS Bedrock: $0.09/GB data out of AWS region. For a workload generating 1TB/month of API responses (large-scale document processing), that is $90/month in egress alone — often missed in initial estimates.
Azure OpenAI: $0.087/GB data out of Azure datacenter. Azure’s “same-region” definition is narrower than expected; cross-zone traffic within a region can incur charges.
Google Vertex AI: $0.08/GB egress. Vertex’s free egress tier (first 1GB/month) is irrelevant at enterprise scale.
Anthropic Direct: Customer bears CDN/WAF egress costs through their own infrastructure.
API Gateway Costs
| Gateway | Cost Model | Typical Monthly Cost (10M requests) |
|---|---|---|
| AWS API Gateway | $3.50/million requests + $0.09/GB | $35 + data transfer |
| Azure APIM | ~$0.70/10K calls (consumption tier) | $700 |
| GCP API Gateway | $3/million requests | $30 |
| Cloudflare Workers (proxy) | $0.15/million + $0.015/100K KV reads | $1.50 base |
For high-volume workloads (>50M requests/month), API gateway cost can exceed $1,500–3,500/month — include in TCO.
Logging and Observability
LLM traces are large. A single Claude Opus call with a 10K-token prompt and 2K-token response generates ~8KB of structured trace data. At 1M calls/day, that is 8GB/day of trace data, or 240GB/month.
| Observability Platform | Storage Cost | Ingestion Cost | Monthly @ 240GB |
|---|---|---|---|
| Datadog | $0.10/GB ingested | Included | $24 + retention |
| Langfuse (self-hosted) | ~$0.02/GB (S3) | Labor for infra | $5 + ~$2K/mo DevOps |
| Arize AI | Custom pricing | — | ~$2,000–5,000/mo enterprise |
| Honeycomb | $130/month base + $0.40/GB | Included | $226 |
SDK and Runtime Overhead
- Cold start latency tax: AWS Lambda cold starts with the Bedrock SDK average 800ms–1.2s. Budget for provisioned concurrency or accept the latency cost.
- SDK version drift: Anthropic’s SDK has averaged 1.2 breaking changes per quarter in 2024–2025. Budget 8–16 hours/quarter of engineering maintenance per SDK in active use.
- Retry library complexity: Production-grade retry logic (exponential backoff, jitter, per-model error handling) takes 2–4 weeks to build correctly. If buying through a managed platform (Bedrock, Vertex), the retry layer is partially handled.
7. Procurement Negotiation Leverage
Committed Use Structures
All three major cloud providers offer significant discounts for committed AI spend. Published prices are rarely what enterprise customers pay.
AWS Bedrock:
- No formal committed use discount for Bedrock inference as of Q1 2026; discounts negotiated through AWS EDP (Enterprise Discount Program)
- EDP commitments of $1M+/year typically yield 10–25% off Bedrock inference
- AWS MAP (Migration Acceleration Program) can offset first-year AI workload costs if migrating from on-prem
- Savings Plans do NOT apply to Bedrock; only EC2/Fargate/Lambda
Azure OpenAI:
- Provisioned Throughput Units (PTUs): pay fixed hourly rate for guaranteed TPM (tokens per minute) capacity; typically 20–40% cheaper than pay-as-you-go for predictable workloads
- Azure Committed Use Discounts: 1-year or 3-year commitments through Azure reservations; 15–30% savings
- Microsoft Azure Consumption Commitment (MACC): counts AI spend toward overall Azure commitment, enabling higher tier discounts elsewhere
- Enterprise Agreement (EA) customers: negotiate AI-specific line items in renewal cycles
Google Vertex AI:
- CUDs (Committed Use Discounts): 1-year or 3-year for Vertex AI; 17–30% discount vs. on-demand
- Google CUDS apply to Vertex dedicated endpoints (not shared inference)
- Sustained Use Discounts (SUDs) apply automatically for Vertex AI on some SKUs after 25% monthly usage
- GCP EDP: $1M+ commitment, 10–15% overall discount applied across eligible Vertex services
Anthropic Direct:
- Volume tiers published; enterprise contracts available for >$100K/month
- Typical enterprise discount: 15–30% off published rates for 12-month commitment
- Claude.ai for Enterprise seats separate from API pricing; bundle negotiation possible
- Anthropic does not currently offer quarterly price-lock guarantees; include price adjustment clauses in contracts
Negotiation Leverage Points
-
Multi-cloud optionality: The most powerful leverage is credible multi-cloud architecture. If you can run Claude on Bedrock OR Anthropic Direct, you have price negotiation leverage with both. Document this optionality explicitly in procurement conversations.
-
Batch workload segregation: Offer to route batch/async workloads to the vendor in exchange for lower pricing on real-time workloads. Vendors value predictable batch throughput.
-
Benchmark data as leverage: Sharing your tokenizer-adjusted benchmark results with a competing vendor — showing their effective price is 18% higher than a competitor — regularly unlocks pricing concessions.
-
Renewal timing: AI model prices have declined 40–70% year-over-year (GPT-4-class models, 2023→2025). Build 12-month price-adjustment clauses into contracts. Never sign a 3-year contract at today’s prices without a renegotiation trigger.
8. Total Value Methodology: Cost Per Quality Point
The Problem With Pure Cost Comparison
A model that costs $2/MTok and answers 70% of questions correctly is worse value than a model that costs $3/MTok and answers 90% correctly, for most enterprise workloads. TCO comparisons without quality normalization select for cheap-and-wrong.
Standardized Quality Baselines
For commodity quality benchmarks, use established evals:
| Benchmark | What It Measures | Enterprise Relevance |
|---|---|---|
| MMLU | Broad knowledge across 57 domains | General-purpose Q&A, policy, compliance |
| HumanEval / MBPP | Python code correctness | Engineering copilot, automation |
| MATH / GSM8K | Mathematical reasoning | Finance, modeling, quantitative analysis |
| DROP | Reading comprehension, arithmetic | Document processing, contract analysis |
| BIG-Bench Hard | Multi-step reasoning | Complex decision support |
Custom Quality Evals (Required for Procurement)
Commodity benchmarks are necessary but not sufficient. Build a domain-specific eval for your workload:
- Sample 200 real production queries that represent high-stakes tasks (where errors are costly)
- Produce gold-standard answers via expert review (legal, compliance, finance SMEs)
- Score model outputs on a 0–3 rubric: 0=wrong/harmful, 1=partially correct, 2=correct but verbose, 3=correct and concise
- Compute quality score = (sum of scores) / (200 × 3) × 100
Cost-Per-Quality-Point Formula
CPQ = effective_monthly_cost / quality_score
Example:
Provider A: $12,000/month effective cost, quality score 78
CPQ_A = $12,000 / 78 = $153.85/quality-point
Provider B: $9,500/month effective cost, quality score 65
CPQ_B = $9,500 / 65 = $146.15/quality-point
Provider B is marginally cheaper per quality-point, but the 13-point quality gap
may drive more human review overhead — factor that into total cost.
Human Review Cost Multiplier
For workloads requiring human-in-the-loop review, lower quality scores drive higher review costs:
total_cost = inference_cost + (review_rate × review_cost_per_item × volume)
where review_rate ≈ (1 - quality_score/100) × review_trigger_threshold
A model with a 15% lower quality score on your workload may require 25–40% more human review, easily erasing a 20% inference cost advantage.
9. Time-to-Value Cost: Onboarding and Migration
Onboarding Cost Components
Onboarding cost is rarely modeled in vendor comparisons. It is real and significant.
| Component | AWS Bedrock | Azure OpenAI | Google Vertex | Anthropic Direct |
|---|---|---|---|---|
| IAM / auth setup | 1–3 days | 1–2 days | 1–2 days | Hours |
| SDK integration | 1–2 weeks | 1–2 weeks | 2–3 weeks | 1–2 weeks |
| Observability pipeline | 1–2 weeks | 1 week | 2 weeks | 3–5 weeks (build from scratch) |
| Security review / VPC | 2–4 weeks | 2–3 weeks | 2–4 weeks | 3–6 weeks |
| Typical total (enterprise) | 6–9 weeks | 5–7 weeks | 7–10 weeks | 8–12 weeks |
| Estimated engineering cost | $40–80K | $35–65K | $50–90K | $65–120K |
Assumes a 4-person implementation team at $150/hour blended rate.
SDK Lock-In Switching Costs
Switching providers after initial integration is not free. Quantify the switching cost before committing:
- OpenAI-compatible APIs (Azure OpenAI, many Bedrock models via the Converse API): switching within this group is 1–2 weeks of re-testing; minimal code changes
- Anthropic → OpenAI-compatible: 3–6 weeks; message format differences, tool-use schema differences, system prompt handling differs
- Vertex Gemini → anything else: 4–8 weeks; Vertex SDK is the most divergent from the OpenAI interface standard
- LangChain/LlamaIndex abstraction layer: Adds 20–40% to initial integration time but reduces switching cost to 1–2 weeks for any downstream provider change; positive ROI after the first migration
Procurement recommendation: Mandate OpenAI-compatible API support as a vendor requirement, or require the vendor to commit to a migration support window (30–60 days of parallel operation) in the contract.
10. Benchmarking Cadence
Why Cadence Matters
AI model pricing is not stable. From Q1 2023 to Q1 2026:
- GPT-4-class model prices declined ~85%
- Claude Opus pricing declined ~60% (Sonnet/Haiku tiers emerged, changing the decision matrix)
- Google Gemini pricing declined ~70% for Flash models
- Llama-class open models moved from experimental to production-grade, adding a new cost floor
A benchmark run done at contract signing may be materially wrong at renewal.
Recommended Review Triggers
| Trigger | Action |
|---|---|
| Major model release (GPT-5, Claude 4, Gemini 3) | Full re-benchmark within 30 days |
| Provider announces price change | Re-run tokenizer-adjusted cost comparison within 2 weeks |
| Monthly spend variance > 15% vs. forecast | Diagnose: model version change, tokenizer update, or workload drift |
| Quarterly | Review CV and latency percentiles; check for silent model version bumps |
| Annual | Full TCO re-evaluation including all hidden cost layers |
Silent Model Version Changes
Providers update models without notice on non-versioned endpoints (e.g., gpt-4o may point to different model weights after a provider update). This can cause:
- Quality regressions (detectable via eval score drift)
- Tokenizer behavior changes (detectable via CV spike)
- Latency changes (detectable via p95 shift)
Mitigation: Always pin to versioned model endpoints in production (e.g., gpt-4o-2024-11-20, claude-opus-4-5-20251101). Treat unversioned endpoints as staging-only.
11. Vendor Scorecard Template: 12-Factor Evaluation Matrix
Rate each factor 1–5. Weights reflect enterprise procurement priorities derived from FinOps practitioner surveys (2025).
| # | Factor | Weight | Scoring Guidance |
|---|---|---|---|
| 1 | Tokenizer-adjusted cost per task | 15% | 5=lowest effective CPT in your workload; 1=highest |
| 2 | Cost variance (CV) | 8% | 5=CV<15%; 4=15–25%; 3=25–35%; 2=35–50%; 1=>50% |
| 3 | Latency (p95 TTFT for your workload) | 10% | 5=<300ms; 4=300–600ms; 3=600ms–1.5s; 2=1.5–3s; 1=>3s |
| 4 | Quality score on domain eval | 15% | 5=85–100; 4=75–84; 3=65–74; 2=55–64; 1=<55 |
| 5 | Cost per quality point (CPQ) | 12% | Computed from factors 1 and 4; 5=best CPQ |
| 6 | TCO completeness (hidden cost ratio) | 8% | 5=hidden costs <10% of L1; 1=hidden costs >40% of L1 |
| 7 | Committed use discount depth | 7% | 5=>30% discount available at your scale; 1=<10% |
| 8 | SLA and uptime guarantees | 8% | 5=99.9%+ with financial remedy; 1=no published SLA |
| 9 | Observability and logging maturity | 6% | 5=native trace export, cost attribution; 1=no native tooling |
| 10 | SDK portability / switching cost | 5% | 5=OpenAI-compatible + migration support; 1=proprietary, no migration path |
| 11 | Security and compliance posture | 8% | 5=SOC2 T2 + HIPAA BAA + data residency + no training on customer data; 1=none documented |
| 12 | Roadmap and model refresh cadence | 8% | 5=clear roadmap, 6-month model cadence, advance notice of deprecations; 1=opaque |
Weighted Score:
total_score = Σ (factor_score_i × weight_i)
Interpretation:
4.0–5.0: Preferred vendor; proceed to contract negotiation
3.0–3.9: Viable with contract risk mitigations
2.0–2.9: Requires significant concessions before award
<2.0: Do not shortlist
Sample Scorecard (Illustrative, Q2 2026)
Scores are illustrative for a mid-market enterprise running mixed summarization + code workloads at 5M requests/month. Your workload will produce different results.
| Factor | Bedrock (Claude) | Azure OpenAI (GPT-4o) | Vertex (Gemini 2.0 Flash) | Anthropic Direct |
|---|---|---|---|---|
| 1. Tokenizer-adjusted cost | 3 | 4 | 5 | 4 |
| 2. Cost CV | 4 | 4 | 3 | 4 |
| 3. Latency p95 | 3 | 4 | 4 | 3 |
| 4. Quality score | 5 | 4 | 3 | 5 |
| 5. CPQ | 4 | 4 | 4 | 4 |
| 6. Hidden cost ratio | 3 | 3 | 4 | 2 |
| 7. Committed discounts | 4 | 5 | 4 | 3 |
| 8. SLA guarantees | 4 | 5 | 4 | 2 |
| 9. Observability | 4 | 5 | 4 | 2 |
| 10. SDK portability | 4 | 5 | 3 | 4 |
| 11. Security/compliance | 5 | 5 | 4 | 4 |
| 12. Roadmap clarity | 3 | 4 | 4 | 4 |
| Weighted Total | 3.75 | 4.22 | 3.78 | 3.38 |
This illustrative scorecard does not constitute a vendor recommendation. Run against your workload, procurement constraints, and negotiated pricing.
Appendix A: Benchmark Execution Checklist
Before running a cost benchmark, verify:
- [ ] Prompt corpus is drawn from production logs, not synthetic data
- [ ] Corpus is stratified across the 5 workload categories (§4)
- [ ] N ≥ 200 prompts per category (1,000 total minimum)
- [ ] Tokenizer counting uses each vendor’s official tokenizer library
- [ ] Benchmark runs at realistic concurrency (not sequential single-thread)
- [ ] Latency measured at p50, p95, p99 (not mean)
- [ ] All runs use versioned model endpoints
- [ ] Results include CV in addition to mean cost
- [ ] Hidden cost layers (egress, gateway, logging) are estimated and included in total
- [ ] Quality eval is run in parallel on the same prompt set
Appendix B: Price Change Velocity Reference
For planning benchmark cadence and contract escalation clauses:
| Period | GPT-4-class price change | Claude-class price change | Gemini-class price change |
|---|---|---|---|
| 2023 → 2024 | -60% | -45% | N/A (launched 2024) |
| 2024 → 2025 | -55% | -50% | -65% |
| 2025 → Q2 2026 | -30% (est.) | -35% (est.) | -40% (est.) |
At these rates, a 3-year price commitment at today’s rates overpays by approximately 60–70% in year 3 relative to spot pricing. Structure contracts with annual price-review clauses tied to published rate cards.
Key Findings for Procurement Decision-Makers
-
Published per-token prices are not comparable. Tokenizer normalization is mandatory and typically shifts effective cost rankings by 10–25% relative to face-value comparisons.
-
Layer 1 (API inference) is 35–50% of 3-year TCO. Optimizing only on published API prices ignores more than half of actual spend.
-
Azure OpenAI scores highest on the 12-factor scorecard for mixed enterprise workloads primarily due to PTU pricing, SLA strength, and native observability integration — not raw token price.
-
Anthropic Direct has the highest quality ceiling but the highest implementation overhead. Appropriate for quality-critical, low-volume workloads or teams with mature MLOps infrastructure.
-
Gemini 2.0 Flash offers the best CPQ for high-volume, lower-stakes summarization workloads. Not competitive for complex reasoning at enterprise quality thresholds.
-
Never sign AI contracts without price-adjustment clauses. Model pricing has declined 60–85% over 3 years. A 3-year lock at current prices is a material financial risk.
-
Build the multi-cloud optionality before negotiating. The ability to credibly route workloads to a competing platform is the single most effective procurement lever.