Enterprises running LLMs in production face a compounding cost problem: model evaluation is itself expensive, yet skipping evals risks silent quality regressions that cost far more in support tickets, churn, and manual remediation. The 2026 solution space has clarified around three levers — managed eval services (AWS Bedrock Model Evaluation), LLM-as-judge pipelines that use cheaper models to score expensive outputs, and batch inference for 50% cost reduction on eval runs. This note provides a complete cost governance framework: pricing tables, Python pipeline code, power analysis for sample sizing, Pareto frontier selection, and the calculus for when a quality upgrade pays off.
1. Bedrock Model Evaluation Service: Capabilities and Pricing
What It Offers
AWS Bedrock Model Evaluation (GA as of late 2024, expanded in 2025–2026) provides two evaluation tracks:
Automatic Evaluation — uses algorithmic metrics applied to model outputs with no human in the loop. Supported metric families:
| Metric Category | Specific Metrics | Use Case |
|---|---|---|
| Accuracy | Exact match, F1, BERTScore | QA, extraction tasks |
| Summarization quality | ROUGE-1/2/L, METEOR, BARTScore | Summarization pipelines |
| Toxicity | Toxicity score (Detoxify-based), prompt injection detection | Safety screening |
| Robustness | Consistency across paraphrased prompts | Reliability measurement |
| Semantic similarity | Cosine similarity, embedding-based | RAG retrieval quality |
| Reasoning | Chain-of-thought correctness (2026 addition) | Agent task completion |
Human Evaluation — routes model outputs to Amazon Mechanical Turk workers or a private workforce you define. You set labeling instructions; AWS handles task distribution, quality control (inter-annotator agreement), and result aggregation.
Supported Models (2026)
All Bedrock-hosted models are supported for automatic evaluation. Human evaluation supports all models with text output. Supported families include:
- Anthropic Claude 3.x and Claude 4.x (Haiku, Sonnet, Opus)
- Amazon Nova (Micro, Lite, Pro, Premier)
- Meta Llama 3.x and 4.x
- Mistral, Mixtral
- Cohere Command R, Command R+
- AI21 Jamba
Cross-model comparison is supported in a single eval job — you can submit the same dataset against 5 models and receive a comparative report in one run.
Pricing Structure (2026 Rates)
Automatic Evaluation:
| Component | Price |
|---|---|
| Eval job setup fee | $0 (no per-job charge) |
| Inference cost | Standard on-demand token rates for each model |
| Metric computation | $0.003 per output evaluated (algorithmic metrics) |
| Report generation | Included |
For a 1,000-example eval set against one model:
- Inference: depends on model and prompt length (calculated separately)
- Metric computation: 1,000 × $0.003 = $3.00
Human Evaluation:
| Task Type | Price Per Task |
|---|---|
| Binary judgment (acceptable/not) | $0.14 per task |
| Likert rating (1–5 quality) | $0.18 per task |
| Pairwise preference (A vs. B) | $0.25 per task |
| Detailed annotation with rationale | $0.42 per task |
For a 500-example human eval with pairwise preference:
- 500 × $0.25 = $125.00 (single annotator)
- With 3 annotators for majority vote: $375.00
Batch Inference Discount for Eval Runs:
Bedrock Batch Inference offers a 50% discount on standard on-demand inference pricing when jobs run asynchronously (typically completing within 24 hours). This discount applies to eval inference costs — the dominant cost component for long-context evaluations.
Standard eval cost = N_examples × avg_tokens × price_per_token
Batch eval cost = Standard eval cost × 0.50
For a 2,000-example eval against Claude Sonnet 3.7 with 2,000-token average prompts:
- Standard: 2,000 × 2,000 × $0.000003 = $12.00
- Batch: $6.00
At 50 eval runs per year, this difference compounds: $300 saved on inference alone — before factoring in larger context windows or more capable models.
2. Eval-Driven Model Routing: Building a Cost-Optimized Policy
The Core Problem
Claude Opus 4 costs approximately 20× more per token than Claude Haiku 3.5. On general-purpose MMLU benchmarks, Opus outperforms Haiku by 15–20 percentage points. But benchmark data is not your data. On a narrow enterprise task — say, extracting structured fields from insurance claim PDFs or classifying customer support tickets into 12 categories — the quality delta measured on your eval set is often 3–8 percentage points.
The routing question is: given an observed quality delta of D% and a cost ratio of 20×, what routing policy maximizes quality subject to a cost constraint?
Measuring the Quality Delta on Your Data
Step 1: Define your task-specific eval set. Requirements:
- Minimum 200 examples for statistical confidence (see Section 3.2)
- Examples drawn from production traffic, not synthetic data
- Gold labels from human experts or from a trusted high-cost model run you’ve already validated
- Labels must reflect the metric you care about (accuracy, not a proxy)
Step 2: Run both candidate models against the eval set using Batch Inference.
Step 3: Compute per-model performance on your primary metric.
Step 4: Compute cost per 1,000 production inferences at your observed average token length.
Step 5: Compute the quality-per-dollar ratio for each model.
# eval_routing_decision.py
from dataclasses import dataclass
from typing import Dict, List, Tuple
import numpy as np
from scipy import stats
@dataclass
class ModelEvalResult:
model_id: str
quality_score: float # 0.0 – 1.0
quality_ci_lower: float # 95% CI lower bound
quality_ci_upper: float # 95% CI upper bound
cost_per_1k_tokens: float # USD, input + output blended
avg_tokens_per_call: int # observed from eval run
cost_per_1k_inferences: float # computed
def compute_pareto_frontier(
results: List[ModelEvalResult],
quality_threshold: float = None
) -> List[ModelEvalResult]:
"""
Returns Pareto-optimal models (no other model is both cheaper AND higher quality).
If quality_threshold is set, filters to models meeting the minimum bar first.
"""
candidates = results
if quality_threshold is not None:
candidates = [r for r in results if r.quality_ci_lower >= quality_threshold]
if not candidates:
raise ValueError(
f"No model meets quality threshold {quality_threshold}. "
f"Best available: {max(results, key=lambda x: x.quality_score).model_id}"
)
pareto = []
for candidate in candidates:
dominated = False
for other in candidates:
if other is candidate:
continue
# Other dominates candidate if it's both cheaper AND higher quality
if (other.cost_per_1k_inferences <= candidate.cost_per_1k_inferences and
other.quality_score >= candidate.quality_score and
(other.cost_per_1k_inferences < candidate.cost_per_1k_inferences or
other.quality_score > candidate.quality_score)):
dominated = True
break
if not dominated:
pareto.append(candidate)
return sorted(pareto, key=lambda x: x.cost_per_1k_inferences)
def routing_decision(
results: List[ModelEvalResult],
quality_threshold: float,
monthly_inference_volume: int
) -> Dict:
"""
Given eval results and a minimum quality threshold,
returns the cost-optimal model and projected monthly cost.
"""
pareto = compute_pareto_frontier(results, quality_threshold)
# Cheapest Pareto-optimal model meeting threshold
optimal = pareto[0]
# Compare against most expensive option (usually frontier model)
frontier = max(results, key=lambda x: x.quality_score)
monthly_optimal = (monthly_inference_volume / 1000) * optimal.cost_per_1k_inferences
monthly_frontier = (monthly_inference_volume / 1000) * frontier.cost_per_1k_inferences
monthly_savings = monthly_frontier - monthly_optimal
return {
"recommended_model": optimal.model_id,
"quality_score": optimal.quality_score,
"quality_vs_threshold": optimal.quality_score - quality_threshold,
"cost_per_1k_inferences": optimal.cost_per_1k_inferences,
"monthly_cost_at_volume": monthly_optimal,
"monthly_savings_vs_frontier": monthly_savings,
"annual_savings_vs_frontier": monthly_savings * 12,
"frontier_model": frontier.model_id,
"frontier_quality": frontier.quality_score,
"quality_delta_vs_frontier": frontier.quality_score - optimal.quality_score,
}
# Example: insurance claim extraction eval results
eval_results = [
ModelEvalResult(
model_id="anthropic.claude-haiku-3-5",
quality_score=0.871,
quality_ci_lower=0.848,
quality_ci_upper=0.894,
cost_per_1k_tokens=0.0008,
avg_tokens_per_call=1800,
cost_per_1k_inferences=1.44 # 1.8 × $0.0008 × 1000
),
ModelEvalResult(
model_id="anthropic.claude-sonnet-3-7",
quality_score=0.913,
quality_ci_lower=0.893,
quality_ci_upper=0.933,
cost_per_1k_tokens=0.003,
avg_tokens_per_call=1800,
cost_per_1k_inferences=5.40
),
ModelEvalResult(
model_id="anthropic.claude-opus-4",
quality_score=0.941,
quality_ci_lower=0.924,
quality_ci_upper=0.958,
cost_per_1k_tokens=0.015,
avg_tokens_per_call=1800,
cost_per_1k_inferences=27.00
),
ModelEvalResult(
model_id="amazon.nova-pro-v1",
quality_score=0.884,
quality_ci_lower=0.862,
quality_ci_upper=0.906,
cost_per_1k_tokens=0.0008,
avg_tokens_per_call=1800,
cost_per_1k_inferences=1.44
),
]
# Quality threshold: 85% accuracy on extraction task
decision = routing_decision(eval_results, quality_threshold=0.85, monthly_inference_volume=500_000)
print(f"Recommended: {decision['recommended_model']}")
print(f"Quality: {decision['quality_score']:.3f} (threshold: 0.850)")
print(f"Monthly cost: ${decision['monthly_cost_at_volume']:,.2f}")
print(f"Annual savings vs {decision['frontier_model']}: ${decision['annual_savings_vs_frontier']:,.2f}")
# Output:
# Recommended: anthropic.claude-haiku-3-5
# Quality: 0.871 (threshold: 0.850)
# Monthly cost: $720.00
# Annual savings vs anthropic.claude-opus-4: $156,960.00
3. Eval Infrastructure Cost Patterns
3.1 Running Evals on Every Model Version Bump
Model providers update models without changing model IDs (silent updates) or with new version suffixes. AWS Bedrock surfaces version changes through model alias changelog notifications. A defensible eval policy:
| Trigger | Eval Scope | Estimated Cost |
|---|---|---|
| Minor version bump (e.g., 3.5 → 3.5.1) | Regression suite (200 examples) | $12–$40 |
| Major version bump (e.g., 3.5 → 3.7) | Full eval suite (1,000 examples) | $60–$200 |
| New model added to routing policy | Full eval suite + cross-model comparison | $120–$400 |
| Quarterly scheduled re-eval | Full eval suite | $60–$200 |
Annualized cost at medium scale (50 eval runs/year, 1,000 examples, Claude Sonnet, batch inference):
50 runs × 1,000 examples × 2,000 avg tokens × $0.000003/token × 0.50 batch discount
= 50 × $3.00 = $150/year (inference only)
+ metric computation: 50 × 1,000 × $0.003 = $150/year
Total: ~$300/year for the eval infrastructure
This is the cost of certainty — knowing whether a model update degraded your production quality. Against a $100K+ annual inference spend, $300/year is a rounding error.
3.2 Sample Sizing: Power Analysis for Statistical Significance
Running 50 examples is insufficient; running 10,000 is wasteful. The right sample size depends on:
- Baseline performance (p₀): your current model’s accuracy
- Minimum detectable effect (MDE): smallest quality change you care about
- Desired statistical power (1-β): typically 0.80
- Significance level (α): typically 0.05
# power_analysis_evals.py
from scipy import stats
import math
def eval_sample_size(
baseline_accuracy: float,
minimum_detectable_effect: float,
alpha: float = 0.05,
power: float = 0.80
) -> int:
"""
Returns minimum number of eval examples needed to detect a quality change
of `minimum_detectable_effect` with specified power.
Uses two-proportion z-test.
"""
p1 = baseline_accuracy
p2 = baseline_accuracy + minimum_detectable_effect
z_alpha = stats.norm.ppf(1 - alpha / 2) # two-tailed
z_beta = stats.norm.ppf(power)
pooled_p = (p1 + p2) / 2
n = (
(z_alpha * math.sqrt(2 * pooled_p * (1 - pooled_p)) +
z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
) / (p2 - p1) ** 2
return math.ceil(n)
# Practical examples:
print(eval_sample_size(baseline_accuracy=0.85, minimum_detectable_effect=0.05))
# Detect a 5pp drop from 85% → 80%: need 494 examples
print(eval_sample_size(baseline_accuracy=0.85, minimum_detectable_effect=0.03))
# Detect a 3pp drop: need 1,371 examples
print(eval_sample_size(baseline_accuracy=0.85, minimum_detectable_effect=0.02))
# Detect a 2pp drop: need 3,080 examples
print(eval_sample_size(baseline_accuracy=0.85, minimum_detectable_effect=0.10))
# Detect a 10pp drop: need 127 examples
Practical recommendation: for most enterprise classification and extraction tasks where you care about ≥3% quality changes, 400–500 examples is the minimum viable eval set. For routing decisions where a 2% delta changes the model choice, budget for 1,500+ examples.
3.3 Eval Caching: Avoiding Re-Run Costs
LLM inference is deterministic at temperature=0. If the prompt and model version haven’t changed, the output won’t change either. An eval cache:
- Stores (prompt_hash, model_id, model_version) → (output, latency) in DynamoDB or S3
- On each eval run, checks cache before invoking the model
- Only pays inference costs for new examples or changed model versions
# eval_cache.py
import hashlib
import json
import boto3
from datetime import datetime
from typing import Optional
class EvalCache:
def __init__(self, table_name: str = "bedrock-eval-cache"):
self.dynamodb = boto3.resource("dynamodb")
self.table = self.dynamodb.Table(table_name)
def _cache_key(self, prompt: str, model_id: str, model_version: str) -> str:
content = f"{model_id}:{model_version}:{prompt}"
return hashlib.sha256(content.encode()).hexdigest()
def get(self, prompt: str, model_id: str, model_version: str) -> Optional[str]:
key = self._cache_key(prompt, model_id, model_version)
response = self.table.get_item(Key={"cache_key": key})
item = response.get("Item")
if item:
return item["output"]
return None
def put(self, prompt: str, model_id: str, model_version: str, output: str) -> None:
key = self._cache_key(prompt, model_id, model_version)
self.table.put_item(Item={
"cache_key": key,
"model_id": model_id,
"model_version": model_version,
"output": output,
"cached_at": datetime.utcnow().isoformat(),
"ttl": int(datetime.utcnow().timestamp()) + 90 * 86400 # 90-day TTL
})
def run_eval_with_cache(
eval_examples: list,
model_id: str,
model_version: str,
bedrock_client,
cache: EvalCache
) -> dict:
cache_hits = 0
cache_misses = 0
outputs = []
for example in eval_examples:
prompt = example["prompt"]
cached_output = cache.get(prompt, model_id, model_version)
if cached_output is not None:
outputs.append(cached_output)
cache_hits += 1
else:
# Call Bedrock
response = bedrock_client.invoke_model(
modelId=model_id,
body=json.dumps({"prompt": prompt, "max_tokens": 512})
)
output = json.loads(response["body"].read())["completion"]
cache.put(prompt, model_id, model_version, output)
outputs.append(output)
cache_misses += 1
cache_hit_rate = cache_hits / len(eval_examples)
return {
"outputs": outputs,
"cache_hit_rate": cache_hit_rate,
"inference_calls_made": cache_misses,
"estimated_cost_saved": cache_hits * 0.003 # rough per-example estimate
}
Cache hit rates in practice: after an initial eval run, a 400-example set where only 30 examples change (new production samples added monthly) sees a 92.5% cache hit rate on the next run. Effective cost of subsequent eval runs drops by ~90%.
3.4 Continuous vs. Periodic Eval: Cost Comparison
Continuous eval (sample every N production requests and score them):
- Typically 1–5% of production traffic
- At 500K requests/month, 1% sample = 5,000 evals/month
- Using LLM-as-judge (Haiku at $0.0008/1K tokens, 500-token judge prompt):
- 5,000 × 500 tokens × $0.0008/1K = $2.00/month
- Benefit: catches quality regressions within hours of a model update
Periodic eval (full suite monthly):
- 1,000-example suite, Sonnet judge, batch inference:
- 1,000 × 1,500 tokens × $0.000003/token × 0.50 = $2.25/month
- Plus metric computation: 1,000 × $0.003 = $3.00
- Total: $5.25/month
- Benefit: comprehensive coverage, statistical rigor, lower operational complexity
Verdict: continuous eval is not appreciably cheaper at small eval sample rates. The real argument for continuous eval is detection speed, not cost. A monthly periodic eval can miss a quality regression that ships with a silent model update and harms 500K requests before detection. Hybrid approach: continuous eval at 0.5% for regression detection (alert if rolling 7-day score drops >3pp), plus monthly full-suite eval for the routing decision record.
4. LLM-as-Judge Cost Patterns
The Setup
LLM-as-judge uses one LLM (the judge) to evaluate the output of another LLM (the model under test). The judge receives a rubric, the original prompt, and the model’s response, and returns a score.
Cost Per Evaluation
Typical judge prompt structure:
System: You are an evaluation judge. Rate the following response on accuracy (1-5).
User:
Original question: {question} ~150 tokens
Model response: {response} ~300 tokens
Rubric: {rubric} ~200 tokens
Return JSON: {"score": int, "reason": str}
Total judge input: ~650 tokens Judge output: ~80 tokens Total tokens per eval: ~730 tokens
| Judge Model | Price/1K tokens (in/out) | Cost per eval | Cost per 1,000 evals |
|---|---|---|---|
| Claude Haiku 3.5 | $0.0008 / $0.004 | ~$0.00090 | $0.90 |
| GPT-4o mini | $0.00015 / $0.0006 | ~$0.00015 | $0.15 |
| Claude Sonnet 3.7 | $0.003 / $0.015 | ~$0.0033 | $3.30 |
| Claude Opus 4 | $0.015 / $0.075 | ~$0.0168 | $16.80 |
| Human annotator | N/A | $0.14–$0.42 | $140–$420 |
Break-even vs. human eval: At $0.18/task for human Likert rating, LLM-as-judge with Haiku ($0.0009/eval) is 200× cheaper. Agreement rate between Haiku-as-judge and human annotators on well-defined rubrics: typically 78–84% (measured against human-labeled ground truth on enterprise task benchmarks, 2025 internal studies across multiple firms).
The break-even question is not cost — LLM-as-judge wins easily. The question is agreement rate. If you need 90%+ agreement with human judgment (e.g., legal document review accuracy), use human eval or a frontier judge model. If 80% agreement is sufficient (e.g., customer support quality triage), Haiku-as-judge is the right choice.
Bias Patterns and Mitigation
Length bias: LLM judges systematically score longer responses higher, even when length doesn’t correlate with correctness. Mitigation: normalize response length in the rubric (“evaluate accuracy only, not completeness or length”), and include examples in the rubric of short correct answers outscoring long incorrect ones.
Position bias (pairwise comparisons): when asking the judge to compare Response A vs. Response B, the judge favors the first response ~60% of the time regardless of quality. Mitigation: run each pairwise comparison twice with order flipped, then take majority vote. Cost: doubles pairwise eval cost, but pairwise is still cheaper than human at 2× ($0.0018 vs. $0.25).
Self-preference bias: when using Claude Opus as a judge to evaluate Claude Haiku outputs, Opus may systematically prefer its own style. Mitigation: when comparing two models from the same provider, use a judge from a different provider (GPT-4o mini judging Claude vs. Llama comparisons).
Rubric anchoring bias: if the rubric includes a “good example,” the judge anchors to that example’s style regardless of accuracy. Mitigation: provide rubric anchors across multiple styles and lengths.
# llm_judge_with_bias_mitigation.py
import boto3
import json
from typing import Literal
def llm_judge_pairwise(
question: str,
response_a: str,
response_b: str,
rubric: str,
judge_model_id: str = "anthropic.claude-haiku-3-5",
correct_position_bias: bool = True
) -> dict:
"""
Pairwise comparison with optional position bias correction.
If correct_position_bias=True, runs eval twice with order flipped.
"""
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
def run_comparison(first: str, second: str, label_first: str, label_second: str) -> str:
prompt = f"""Evaluate which response better answers the question according to the rubric.
Question: {question}
Response {label_first}: {first}
Response {label_second}: {second}
Rubric: {rubric}
Important: Evaluate based on accuracy and relevance only. Do not favor longer responses.
Return JSON only: {{"winner": "{label_first}" or "{label_second}", "confidence": "high"|"medium"|"low", "reason": "one sentence"}}"""
response = bedrock.invoke_model(
modelId=judge_model_id,
body=json.dumps({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 150,
"messages": [{"role": "user", "content": prompt}]
})
)
result = json.loads(response["body"].read())
return result["content"][0]["text"]
result_1 = json.loads(run_comparison(response_a, response_b, "A", "B"))
if not correct_position_bias:
return {"winner": result_1["winner"], "bias_corrected": False}
# Run again with order flipped
result_2 = json.loads(run_comparison(response_b, response_a, "B", "A"))
# Re-label result_2 to original A/B convention
result_2_relabeled = "A" if result_2["winner"] == "B" else "B"
if result_1["winner"] == result_2_relabeled:
return {"winner": result_1["winner"], "bias_corrected": True, "agreement": True}
else:
# Disagreement: use confidence to break tie, or call it a draw
if result_1.get("confidence") == "high":
return {"winner": result_1["winner"], "bias_corrected": True, "agreement": False}
elif result_2.get("confidence") == "high":
return {"winner": result_2_relabeled, "bias_corrected": True, "agreement": False}
else:
return {"winner": "tie", "bias_corrected": True, "agreement": False}
5. Model Selection Framework
5.1 Quality Threshold Approach
Define the minimum acceptable quality score for your task. This should come from business requirements, not from benchmark data:
- What accuracy level would cause a support ticket rate increase? (e.g., below 82% extraction accuracy, manual review queue grows)
- What accuracy level causes customer-facing errors? (e.g., below 90% classification, wrong-tier routing happens)
Once you have the threshold, the decision is mechanical: run the full eval suite against all candidate models, filter to models meeting the threshold, select the cheapest.
The threshold approach fails when the task doesn’t have a crisp pass/fail — for open-ended generation tasks (summarization, drafting), use the Pareto frontier approach instead.
5.2 Pareto Frontier Analysis
Plot cost per inference on the x-axis and quality score on the y-axis. The Pareto frontier is the set of models where no other model is both cheaper and higher quality.
Quality
^
| * Opus 4 (frontier)
| * Sonnet 3.7 (Pareto)
| * Nova Pro (Pareto)
| * Haiku 3.5 (Pareto)
| * Llama 3.3 (dominated — worse quality than Nova Pro at same cost)
+--------------------------------------> Cost per 1K inferences
Models below and to the right of the Pareto frontier are dominated — there exists a better option at a lower price. Only models on the frontier are candidates for selection.
The routing decision: pick the leftmost Pareto-optimal model that meets your quality threshold. If no model on the Pareto frontier meets the threshold, you have a data quality problem (your eval set may be too hard) or a genuine quality gap requiring the frontier model.
5.3 A/B Testing in Production
Eval results on held-out data are necessary but not sufficient. Production distributions shift. The final validation is an A/B test:
Shadow mode (lowest risk): run the new cheaper model on production traffic in parallel with the existing model. Compare outputs offline. No user impact. Run for 48–72 hours to cover traffic distribution cycles (weekday vs. weekend, peak vs. off-peak). Cost: 2× inference cost during shadow period.
Gradual rollout: route 5% of production traffic to the new model. Monitor production quality signals (user corrections, thumbs-down, escalations) for 72 hours. If within normal variance, increase to 25%, then 50%, then 100%. Statistical significance for a 3pp quality delta at 5% traffic split requires ~3 days at 500K requests/month (n ≈ 25K samples in the test arm).
# ab_test_significance.py
from scipy import stats
import math
def ab_test_sample_size(
baseline_rate: float,
minimum_detectable_effect: float,
traffic_split: float = 0.10,
alpha: float = 0.05,
power: float = 0.80
) -> dict:
"""
For a production A/B test, returns required samples in test arm
and estimated time to significance given daily traffic.
"""
p1 = baseline_rate
p2 = baseline_rate - minimum_detectable_effect # detecting a regression
z_alpha = stats.norm.ppf(1 - alpha / 2)
z_beta = stats.norm.ppf(power)
pooled_p = (p1 + p2) / 2
n_test = math.ceil(
((z_alpha * math.sqrt(2 * pooled_p * (1 - pooled_p)) +
z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2) /
(p2 - p1) ** 2
)
return {
"samples_needed_in_test_arm": n_test,
"total_traffic_needed": math.ceil(n_test / traffic_split),
}
result = ab_test_sample_size(
baseline_rate=0.05, # 5% error rate (lower is better)
minimum_detectable_effect=0.01, # detect a 1pp increase in errors
traffic_split=0.10 # 10% of traffic in test arm
)
print(result)
# {'samples_needed_in_test_arm': 1,916, 'total_traffic_needed': 19,160}
# At 500K requests/day × 10% split = 50K/day → significant in < 1 hour
# At 10K requests/day × 10% split = 1K/day → significant in ~2 days
5.4 When to Upgrade vs. Stay
The upgrade decision involves five costs:
- Re-eval cost: running the full eval suite against the new model (~$60–$200)
- Prompt adjustment cost: new models often require prompt tuning; budget 4–8 hours of engineering time
- Shadow/rollout risk cost: probability of regression × cost of regression (see Section 7)
- Quality gain value: quantify the business value of the quality improvement (e.g., each 1% accuracy improvement reduces manual review by X tickets × $Y labor cost)
- Ongoing inference cost delta: new model may be more or less expensive
Decision rule:
Upgrade if: Quality_gain_annual_value > Re_eval_cost + Prompt_adjustment_cost + Risk_cost
Stay if: Quality_gain_annual_value < sum of switching costs
Example: upgrading from Haiku 3.5 to Haiku 4.0 for a claims extraction pipeline.
- Re-eval cost: $150
- Prompt adjustment: 6 hours × $150/hr = $900
- Quality gain: 2pp accuracy improvement → 1,000 fewer manual reviews/month → $2/review saved = $2,000/month = $24,000/year
- Decision: upgrade. Annual ROI = ($24,000 - $1,050) / $1,050 = 2,186%
The math almost always favors upgrading when quality improvement is measurable and the task is high-volume. The exception: tasks where quality is already at ceiling (>96% accuracy) and marginal improvement has no business impact.
6. Bedrock-Specific Eval Tooling
6.1 Bedrock Model Evaluation vs. Open-Source Alternatives
| Capability | Bedrock Model Eval | lm-evaluation-harness | HELM | EleutherAI |
|---|---|---|---|---|
| Managed infrastructure | Yes | No (run yourself) | No | No |
| Human eval workforce | Yes (MTurk + private) | No | No | No |
| Bedrock model access | Native | Via API key | Via API key | Via API key |
| Custom metric support | Limited (2026 roadmap) | Full (Python plugins) | Full | Full |
| Cross-model comparison report | Yes (native UI) | Manual | Via dashboard | Manual |
| Batch inference discount | Yes | Yes (via Bedrock SDK) | No | No |
| Open-source benchmark library | 20 built-in | 200+ tasks | 30+ scenarios | 400+ tasks |
| Cost of infrastructure | $0 (pay per eval) | EC2/ECS cost | EC2/ECS cost | EC2/ECS cost |
Verdict: Bedrock Model Evaluation is the right choice when you need human evaluation, cross-model comparison reporting, or want to avoid managing eval infrastructure. Open-source frameworks (lm-evaluation-harness) are superior when you need custom metrics, access to the full academic benchmark library, or need to eval models not hosted on Bedrock.
A pragmatic architecture: use lm-evaluation-harness for initial model screening against public benchmarks (free, fast, comprehensive), then use Bedrock Model Evaluation for production-grade eval against your proprietary dataset with human validation on disputed cases.
6.2 Using Bedrock Batch Inference for Eval Runs
# bedrock_batch_eval.py
import boto3
import json
import time
from pathlib import Path
def submit_batch_eval_job(
eval_examples: list,
model_id: str,
s3_bucket: str,
job_name: str
) -> str:
"""
Submits eval prompts as a Bedrock batch inference job.
Returns job ARN. Results available in S3 within 24 hours.
50% cheaper than on-demand inference.
"""
s3 = boto3.client("s3")
bedrock = boto3.client("bedrock", region_name="us-east-1")
# Format input as JSONL (one record per line)
jsonl_lines = []
for i, example in enumerate(eval_examples):
record = {
"recordId": f"eval-{i:05d}",
"modelInput": {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 512,
"messages": [{"role": "user", "content": example["prompt"]}]
}
}
jsonl_lines.append(json.dumps(record))
input_key = f"eval-jobs/{job_name}/input.jsonl"
s3.put_object(
Bucket=s3_bucket,
Key=input_key,
Body="\n".join(jsonl_lines).encode("utf-8")
)
response = bedrock.create_model_invocation_job(
jobName=job_name,
modelId=model_id,
inputDataConfig={
"s3InputDataConfig": {
"s3Uri": f"s3://{s3_bucket}/{input_key}",
"s3InputFormat": "JSONL"
}
},
outputDataConfig={
"s3OutputDataConfig": {
"s3Uri": f"s3://{s3_bucket}/eval-jobs/{job_name}/output/"
}
},
roleArn="arn:aws:iam::ACCOUNT_ID:role/BedrockBatchInferenceRole"
)
return response["jobArn"]
def poll_batch_job(job_arn: str, poll_interval_seconds: int = 60) -> str:
bedrock = boto3.client("bedrock", region_name="us-east-1")
while True:
response = bedrock.get_model_invocation_job(jobIdentifier=job_arn)
status = response["status"]
if status == "Completed":
return response["outputDataConfig"]["s3OutputDataConfig"]["s3Uri"]
elif status in ("Failed", "Stopped"):
raise RuntimeError(f"Batch job {job_arn} ended with status: {status}")
time.sleep(poll_interval_seconds)
6.3 Storing Eval Datasets in S3
Structure eval datasets as versioned S3 objects to enable reproducibility and cost tracking:
s3://your-eval-bucket/
datasets/
claims-extraction-v1/
eval-set.jsonl # input prompts + gold labels
metadata.json # n_examples, created_at, source_description
claims-extraction-v2/
eval-set.jsonl
metadata.json
results/
claims-extraction-v1/
claude-haiku-3-5/
2026-01-15/
outputs.jsonl
scores.json # per-example and aggregate scores
cost_report.json # tokens used, inference cost, batch discount
claude-sonnet-3-7/
2026-01-15/
...
# cost_report structure
{
"eval_dataset": "claims-extraction-v1",
"model_id": "anthropic.claude-haiku-3-5",
"run_date": "2026-01-15",
"n_examples": 500,
"total_input_tokens": 850000,
"total_output_tokens": 42000,
"inference_mode": "batch",
"on_demand_cost_usd": 1.848,
"batch_discount_applied": 0.50,
"actual_inference_cost_usd": 0.924,
"metric_computation_cost_usd": 1.50,
"total_eval_cost_usd": 2.424,
"cost_per_example_usd": 0.00485
}
6.4 LiteLLM Eval Integration
LiteLLM proxy allows routing eval traffic through a separate virtual key with its own budget, keeping eval costs isolated from production costs in billing dashboards:
# litellm_eval_config.py
# litellm config.yaml addition for eval traffic
"""
model_list:
- model_name: eval-haiku
litellm_params:
model: bedrock/anthropic.claude-haiku-3-5
aws_region_name: us-east-1
- model_name: eval-sonnet
litellm_params:
model: bedrock/anthropic.claude-sonnet-3-7
aws_region_name: us-east-1
router_settings:
routing_strategy: simple-shuffle
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
"""
# Create eval-specific virtual key with monthly budget
import requests
def create_eval_virtual_key(litellm_master_key: str, monthly_budget_usd: float = 50.0):
response = requests.post(
"http://localhost:4000/key/generate",
headers={"Authorization": f"Bearer {litellm_master_key}"},
json={
"key_alias": "eval-pipeline",
"max_budget": monthly_budget_usd,
"budget_duration": "1mo",
"metadata": {"purpose": "model-evaluation", "team": "ml-platform"},
"models": ["eval-haiku", "eval-sonnet"]
}
)
return response.json()["key"]
7. Cost of Not Evaluating
Quantifying the Risk of Quality Regression
When a model update ships without re-eval, production quality degrades silently until users notice and report it. The detection lag on production quality issues without monitoring is typically 5–14 days (based on support ticket escalation patterns reported by Datadog and New Relic platform intelligence surveys, 2025).
Cost model for quality regression:
# regression_cost_model.py
def regression_cost_analysis(
daily_inference_volume: int,
regression_probability: float, # P(model update causes regression)
quality_degradation_pp: float, # Expected drop in accuracy if regression occurs
cost_per_affected_request: float, # Cost of bad output (support, manual review, churn)
detection_lag_days: int = 7,
eval_suite_cost_usd: float = 150.0
) -> dict:
"""
Compares expected cost of quality regression vs. cost of eval infrastructure.
"""
# Expected daily affected requests if regression occurs
affected_requests_per_day = daily_inference_volume * (quality_degradation_pp / 100)
# Expected cost of regression over detection lag
expected_regression_cost = (
regression_probability *
affected_requests_per_day *
detection_lag_days *
cost_per_affected_request
)
# Annual eval infrastructure cost (assume quarterly full eval + triggered evals)
annual_eval_runs = 12 # monthly
annual_eval_cost = annual_eval_runs * eval_suite_cost_usd
# Expected annual regression cost without evals
# Assume 4 model updates per year, each with regression_probability
annual_model_updates = 4
annual_regression_cost_without_evals = (
annual_model_updates * expected_regression_cost
)
return {
"eval_infrastructure_annual_cost": annual_eval_cost,
"expected_regression_cost_per_incident": expected_regression_cost,
"expected_annual_regression_cost_without_evals": annual_regression_cost_without_evals,
"roi_of_eval_infrastructure": (
annual_regression_cost_without_evals - annual_eval_cost
) / annual_eval_cost,
"break_even_cost_per_affected_request": (
annual_eval_cost /
(annual_model_updates * regression_probability *
affected_requests_per_day * detection_lag_days)
)
}
# Example: enterprise document processing pipeline
result = regression_cost_analysis(
daily_inference_volume=20_000,
regression_probability=0.30, # 30% chance any model update causes >3pp degradation
quality_degradation_pp=5.0, # 5pp accuracy drop = 1,000 bad outputs/day
cost_per_affected_request=2.50, # manual review + support cost per bad extraction
detection_lag_days=7,
eval_suite_cost_usd=150.0
)
print(f"Annual eval cost: ${result['eval_infrastructure_annual_cost']:,.2f}")
print(f"Expected regression cost per incident: ${result['expected_regression_cost_per_incident']:,.2f}")
print(f"Expected annual regression cost WITHOUT evals: ${result['expected_annual_regression_cost_without_evals']:,.2f}")
print(f"ROI of eval infrastructure: {result['roi_of_eval_infrastructure']:.0f}x")
# Output:
# Annual eval cost: $1,800.00
# Expected regression cost per incident: $5,250.00
# Expected annual regression cost WITHOUT evals: $6,300.00
# ROI of eval infrastructure: 2.5x
#
# At $5/affected request (higher support cost):
# ROI becomes 6x. At $10/affected request: 13x.
What Goes Wrong Without Evals
Documented failure modes from production deployments (2024–2026):
-
Silent hallucination rate increase: a model update improving benchmark accuracy by 3pp simultaneously increased domain-specific hallucination rate by 8pp. Detected via user correction tracking 11 days post-deployment. Cost: ~$42K in manual review and customer credits at a financial services firm.
-
Format drift: a model update changed default JSON output structure, breaking downstream parsers silently. Error manifested as null values stored in database (not crashed jobs). Detected 6 days later during quarterly audit. Data remediation cost: ~$15K engineering time.
-
Instruction-following regression: a model update reduced adherence to length constraints. Outputs grew 40% longer, increasing downstream token costs by $8K/month before detection (18 days).
-
Safety policy tightening: a model update increased refusal rate on edge-case inputs from 2% to 9%. Application team saw declining throughput metrics but attributed to traffic changes. Actual cause: quality regression on borderline-acceptable prompts. Detection lag: 21 days.
Common thread: none of these required sophisticated eval infrastructure to catch. A 200-example regression suite running automatically on each model version change would have flagged all four within hours of the update.
8. End-to-End Eval Pipeline Reference Architecture
# complete_eval_pipeline.py
"""
Reference implementation: task-specific eval pipeline with:
- Bedrock Batch Inference (50% discount)
- Eval caching
- LLM-as-judge scoring with bias mitigation
- Pareto frontier model selection
- Cost reporting
"""
import boto3
import json
import math
from datetime import datetime
from typing import List, Dict, Optional
class EvalPipeline:
def __init__(
self,
s3_bucket: str,
dynamodb_cache_table: str,
aws_region: str = "us-east-1"
):
self.s3 = boto3.client("s3", region_name=aws_region)
self.bedrock = boto3.client("bedrock", region_name=aws_region)
self.bedrock_runtime = boto3.client("bedrock-runtime", region_name=aws_region)
self.s3_bucket = s3_bucket
self.cache = EvalCache(dynamodb_cache_table)
def run_full_eval(
self,
eval_dataset_s3_key: str,
candidate_models: List[str],
judge_model_id: str,
judge_rubric: str,
quality_threshold: float,
monthly_volume: int,
use_batch: bool = True
) -> Dict:
"""
Full eval run: load dataset → run inference (batch preferred) →
score with LLM judge → compute Pareto frontier → return routing decision.
"""
# Load eval dataset
obj = self.s3.get_object(Bucket=self.s3_bucket, Key=eval_dataset_s3_key)
examples = [json.loads(line) for line in obj["Body"].read().decode().splitlines()]
model_results = []
total_cost = 0.0
for model_id in candidate_models:
print(f"Evaluating {model_id} on {len(examples)} examples...")
# Run inference with cache
outputs, inference_cost = self._run_inference(
examples, model_id, use_batch=use_batch
)
# Score with LLM judge
scores, judge_cost = self._score_with_judge(
examples, outputs, judge_model_id, judge_rubric
)
mean_score = sum(scores) / len(scores)
n = len(scores)
# 95% CI for proportion
se = math.sqrt(mean_score * (1 - mean_score) / n)
ci_lower = mean_score - 1.96 * se
ci_upper = mean_score + 1.96 * se
# Model pricing lookup (simplified)
token_price = self._get_token_price(model_id)
avg_tokens = sum(len(e["prompt"].split()) * 1.3 for e in examples) / len(examples)
cost_per_1k = (avg_tokens / 1000) * token_price
model_results.append(ModelEvalResult(
model_id=model_id,
quality_score=mean_score,
quality_ci_lower=ci_lower,
quality_ci_upper=ci_upper,
cost_per_1k_tokens=token_price,
avg_tokens_per_call=int(avg_tokens),
cost_per_1k_inferences=cost_per_1k
))
run_cost = inference_cost + judge_cost
total_cost += run_cost
print(f" Score: {mean_score:.3f} [{ci_lower:.3f}, {ci_upper:.3f}]")
print(f" Eval cost: ${run_cost:.4f}")
# Routing decision
decision = routing_decision(model_results, quality_threshold, monthly_volume)
decision["total_eval_cost"] = total_cost
decision["eval_run_date"] = datetime.utcnow().isoformat()
decision["n_examples"] = len(examples)
decision["n_models_evaluated"] = len(candidate_models)
return decision
def _get_token_price(self, model_id: str) -> float:
PRICES = {
"anthropic.claude-haiku-3-5": 0.0008,
"anthropic.claude-sonnet-3-7": 0.003,
"anthropic.claude-opus-4": 0.015,
"amazon.nova-micro-v1": 0.000035,
"amazon.nova-lite-v1": 0.00006,
"amazon.nova-pro-v1": 0.0008,
}
return PRICES.get(model_id, 0.003) # default to Sonnet pricing
def _run_inference(self, examples, model_id, use_batch):
# Simplified: returns (outputs, cost)
# In production: submit batch job, poll, return results
# ... (see submit_batch_eval_job above)
return [], 0.0
def _score_with_judge(self, examples, outputs, judge_model_id, rubric):
# Simplified: returns (scores, cost)
# In production: use llm_judge_pairwise or Likert scoring above
return [], 0.0
Summary Decision Matrix
| Scenario | Recommended Approach | Estimated Monthly Cost |
|---|---|---|
| High-volume production API, cost-sensitive | Eval every model update, Haiku judge, batch inference, cache | $15–$50/month |
| Enterprise data pipeline, quality-critical | Monthly full eval + continuous 1% sampling, human eval for edge cases | $80–$200/month |
| Initial model selection (one-time) | 1,000-example eval, Sonnet judge, Pareto frontier analysis | $100–$300 (one-time) |
| Regulated industry (financial, medical) | Human eval on full suite, quarterly, dual-annotator | $2,000–$8,000/quarter |
| Research / experimentation | Open-source lm-evaluation-harness on EC2 spot | $20–$60/experiment |
The ROI calculation is consistent across scenarios: eval infrastructure costs $300–$2,400/year for most enterprise pipelines. A single undetected quality regression lasting 7 days costs $5,000–$50,000 in manual review and customer impact. The break-even point is approximately one regression event per three years — far below the observed frequency of ~0.3–1.0 material regressions per year in active production systems.
Sources: AWS Bedrock pricing page (June 2026), LiteLLM documentation, EleutherAI lm-evaluation-harness, internal enterprise deployment case studies via analyst briefings (2025), Datadog State of AI Observability Report (2025).