← Agent Frameworks 🕐 17 min read

Agent Frameworks

Bedrock Model Evaluation Cost Governance (2026)

Executive Summary · March 2026

Amazon Bedrock's model evaluation surface spans three cost buckets that require separate governance disciplines: (1) inference charges on the judge model — billed identically to production inference b

Amazon Bedrock’s model evaluation surface spans three cost buckets that require separate governance disciplines: (1) inference charges on the judge model — billed identically to production inference but often overlooked in budget forecasts; (2) human-review task fees ($0.21/task via Bedrock Evaluations, $0.02–$0.08/object via A2I) that scale non-linearly with dataset size; and (3) storage and data-transfer overhead on S3 eval datasets that accumulates invisibly across nightly runs. Without explicit FinOps tagging, all three buckets collapse into the same CUR line items as production inference and become impossible to attribute. This note covers the full pricing structure, CUR discrimination patterns, and a pipeline architecture that keeps eval spend visible, bounded, and auditable.

1. Bedrock Model Evaluation Pricing — Exact Structure

1.1 Automatic Evaluation (Algorithmic Metrics)

Automatic evaluations using built-in algorithmic metrics — accuracy, robustness, toxicity, stereotype detection, factual knowledge — are billed at $0 for the scoring layer. The compute for metrics such as ROUGE, BERTScore, and perplexity is absorbed by AWS. The only charge is the inference consumed to generate the model’s output from the evaluation dataset, billed at standard on-demand rates for whatever model is under evaluation.

Implication: if you run automatic eval on a 1,000-row dataset and the average prompt+completion is 800 tokens, your entire cost is 1,000 × 800 tokens × model on-demand rate. For Claude Haiku 4.5 at $1.00/$5.00 per million input/output, a balanced 500/300 token split costs roughly $0.65 for 1,000 samples — effectively free at small dataset sizes.

1.2 LLM-as-Judge Evaluation

When you select an LLM judge (any Bedrock-supported model) for custom or rubric-based scoring, both the model under evaluation and the judge model are billed at standard on-demand token rates. There is no evaluation surcharge — you pay purely for the tokens consumed.

Token overhead anatomy for a judge invocation:

System prompt / rubric: 200–800 tokens depending on rubric verbosity (stable across runs — prime for caching)
Original prompt from dataset: passes through verbatim
Model response under evaluation: full completion length
Judge output: typically 100–400 tokens (score + brief rationale)

For a 1,000-sample eval where each sample requires a judge call with 1,200 input tokens and 200 output tokens, using Claude Sonnet 4.6 as judge ($3.00/$15.00 per million):

Input cost: 1,000 × 1,200 / 1,000,000 × $3.00 = $3.60
Output cost: 1,000 × 200 / 1,000,000 × $15.00 = $3.00
Total judge cost: ~$6.60 per 1,000 samples

The same run with Haiku 4.5 ($1.00/$5.00):

Input: 1,000 × 1,200 / 1,000,000 × $1.00 = $1.20
Output: 1,000 × 200 / 1,000,000 × $5.00 = $1.00
Total: ~$2.20 per 1,000 samples — 67% cheaper than Sonnet as judge

1.3 Human Evaluation via Bedrock Evaluations

Bedrock’s native human evaluation workflow charges $0.21 per completed human task, where one task equals one human worker reviewing one prompt-response pair and submitting a rating. This fee is additive on top of the inference charge for generating the response.

At 500 items requiring human review:

Human task fees: 500 × $0.21 = $105
Plus inference to generate responses from the eval dataset
Plus any A2I worker marketplace charges if routing to Mechanical Turk

Human eval scales fast. A 5,000-item human eval with a single reviewer per item costs $1,050 in task fees alone — before inference. For large-scale human baseline collection, this is the primary cost driver and must be pre-approved in the budget.

1.4 RAG / Knowledge Base Evaluation

RAG evaluation incurs:

Standard Bedrock Knowledge Base retrieval charges (approximately $0.10 per 1,000 retrieve calls for vector search)
Generator model inference charges (the model producing the response from retrieved context)
Evaluator model charges (judge model scoring the retrieval quality and answer faithfulness)

The retrieval dimension is often underestimated. A 1,000-sample RAG eval making 5 retrieval calls per sample = 5,000 KB retrieve calls = ~$0.50 in retrieval alone, before any inference.

2. CUR Line Items for Model Evaluations

2.1 How Eval Charges Surface in CUR

Bedrock model evaluation does not produce a distinct top-level service in CUR. All charges appear under AmazonBedrock as line_item_product_code. The discrimination happens at the operation level, introduced via the January 2026 granular operation visibility update.

Key CUR columns for eval cost discrimination:

Column	Eval Pattern	Production Pattern
`line_item_product_code`	`AmazonBedrock`	`AmazonBedrock`
`line_item_operation`	`InvokeModelInference`	`InvokeModelInference`
`line_item_usage_type`	`{region}-{ModelID}-Input-Tokens`	`{region}-{ModelID}-Input-Tokens`
`resource_tags_user_environment`	`eval` (if tagged via AIP)	`prod`
`resource_tags_user_workload`	`nightly-eval` (if tagged)	`rag-api`
`line_item_resource_id`	inference profile ARN	inference profile ARN

The critical insight: without application inference profiles (AIPs) and explicit cost allocation tags, eval and production traffic are indistinguishable in CUR. Both show InvokeModelInference in line_item_operation. The only discrimination lever is the AIP ARN in line_item_resource_id — which only works if eval jobs use a dedicated AIP.

2.2 Human Task Fees in CUR

The $0.21/task human evaluation charge surfaces differently:

line_item_product_code: AmazonBedrock
line_item_operation:    HumanEvaluation
line_item_usage_type:   {region}-HumanEvalTask
line_item_description:  "Amazon Bedrock Human Evaluation Task"
pricing_unit:           Tasks

This is one of the few Bedrock CUR entries denominated in Tasks rather than tokens — making it easy to filter and budget separately.

2.3 A2I Human Review Costs in CUR

Amazon A2I charges appear under a separate product code:

line_item_product_code: AmazonA2I
line_item_operation:    HumanReview
line_item_usage_type:   {region}-CustomTaskObjects
pricing_unit:           Objects

A2I charges are fully decoupled from Bedrock charges in CUR. A human review that combines Bedrock inference (to generate the completion for review) and A2I (for the review workflow itself) produces two separate CUR line items under different product codes. Budget owners need to aggregate across both to get true human-review cost.

2.4 S3 Charges for Eval Datasets

Eval dataset storage surfaces under AmazonS3:

line_item_usage_type:  {region}-TimedStorage-ByteHrs
line_item_operation:   StandardStorage

The eval-specific S3 charges are not automatically tagged. Without a resource tag applied to the eval S3 bucket (e.g., purpose: eval-datasets), these costs are invisible in CUR. At $0.023/GB-month for S3 Standard, a 10 GB eval corpus costs $0.23/month — negligible individually, but nightly eval runs can accumulate output JSONL files at scale. A 365-run year with 1 GB of output per run accumulates 365 GB of output data (~$8.40/month if not lifecycle-managed).

3. Bedrock Evaluations API vs Custom Eval Pipelines

3.1 Bedrock Native Evaluations

The Bedrock Evaluations API (CreateEvaluationJob) handles the full eval loop: dataset ingestion from S3, inference invocation, scoring (algorithmic or LLM-as-judge), and results writing back to S3. It supports:

Built-in metrics: accuracy, robustness, toxicity, stereotyping, factual knowledge
Custom metrics via LLM-as-judge with user-provided rubrics
RAG system evaluation (retrieval quality + answer faithfulness)
AgentCore Evaluations for multi-turn agent trajectories

Cost advantage: No orchestration overhead — you pay only for tokens consumed. No Lambda, no Step Functions, no ECS.

Cost disadvantage: Limited caching control. The API does not expose prompt cache configuration for the judge model invocations, so you cannot explicitly mark rubric prefixes as cacheable. This means repeated eval runs over different datasets with the same rubric may re-pay for rubric tokens each time.

Best fit: Teams running infrequent, large-batch evals where orchestration complexity is a burden and the dataset is diverse enough that caching would not help much anyway.

3.2 Custom Eval Pipelines

A custom pipeline using the Bedrock Converse or InvokeModel API directly grants full control:

Explicit cache_control breakpoints on rubric + system prompt (90% input savings on cache hits)
Batch API submission for 50% token cost reduction
Model routing (cheap judge for simple criteria, expensive judge only for edge cases)
Per-request tagging via AIP ARNs
Custom retry logic and partial-run resumption to avoid re-processing completed samples

Cost overhead: Lambda or ECS orchestration, Step Functions for fan-out, CloudWatch alarms — typically $5–$15/month for a nightly eval pipeline at moderate scale.

Best fit: Teams running evals daily or on CI triggers where prompt caching ROI is real and cost per run must be minimized.

3.3 Decision Table

Factor	Bedrock Native API	Custom Pipeline
Rubric reuse across runs	No cache control	Full cache control
Batch discount available	No (sync invocation)	Yes (50% off)
AIP tagging granularity	Job-level	Per-request
Orchestration cost	$0	$5–$15/month
CI/CD integration complexity	Medium (poll job status)	Low (direct SDK calls)
Human eval routing	Native, seamless	Requires A2I SDK integration
Break-even dataset size	< 500 samples/run	> 500 samples/run

4. LLM-as-Judge Cost Patterns

4.1 Token Overhead Anatomy

A judge invocation for a typical LLM-as-judge eval contains four token classes:

Rubric + system prompt (200–800 tokens): Evaluation criteria, scoring scale, output format instructions. Identical across all items in a run — the prime caching target.
Original user prompt (50–500 tokens): The input from the eval dataset.
Model response under evaluation (100–2,000 tokens): The completion generated by the model being evaluated.
Judge output (100–400 tokens): Score and brief rationale; constitutes 100% of output tokens billed.

For a rubric-heavy eval (800-token rubric) scoring 2,000-token completions with a 300-token judge output, total input tokens per judge call ≈ 3,100. At Sonnet 4.6 on-demand rates, that is $0.0093 per sample. At 1,000 samples, $9.30 in judge inference alone.

4.2 Prompt Caching for Judge Prompts

Eval prompts have exceptionally stable prefixes — the rubric, scoring instructions, and output format are byte-identical across every item in a run. This makes them among the highest-ROI prompt caching targets in any Bedrock workload.

How caching applies: Prefix the judge system prompt and rubric with a cache_control: {"type": "ephemeral"} breakpoint. On first invocation, the rubric tokens are written to cache (billed at 1.25× input price for the write). On subsequent calls within the 5-minute cache TTL, those tokens are billed at 0.10× input price.

Expected savings calculation (Sonnet 4.6 judge, 800-token rubric, 1,000 samples in a single run):

Without caching: 1,000 × 800 / 1,000,000 × $3.00 = $2.40 in rubric input tokens
With caching (1 write + 999 reads):
- Write: 1 × 800 / 1,000,000 × $3.75 = $0.003
- Reads: 999 × 800 / 1,000,000 × $0.30 = $0.24
- Total: $0.243 vs $2.40 — 90% reduction on rubric tokens

For a nightly run of 5,000 samples, prompt caching on the rubric prefix saves approximately $10–$12/run depending on rubric length. Over 30 nights, that is $300–$360 in annual savings per eval suite — meaningful when operating multiple eval suites.

Critical constraint: Cache TTL is 5 minutes on Anthropic-hosted models via Bedrock. For large eval runs exceeding 5-minute processing windows, cache will miss on later batches unless the run is structured to process items in rapid succession (parallel workers, not sequential).

4.3 Cache Architecture for Eval Pipelines

To maximize cache hit rates within a single eval run:

Parallelise judge calls with up to 10 concurrent requests (respects standard Bedrock rate limits)
Process items in rapid bursts so all calls land within the 5-minute TTL window
Structure the judge prompt so the rubric occupies the beginning — cache breakpoint immediately after the rubric, before the per-item content
Use a single Bedrock region for all judge calls in a run (cache is region-scoped)

5. Human Review via A2I

5.1 Pricing Structure

Amazon A2I pricing for custom workflows: $0.02–$0.08 per human-reviewed object. This range reflects the worker pool:

Private workforce (your own employees): $0.02–$0.04/object (platform fee only; you pay employee wages separately)
AWS Marketplace vendors: $0.04–$0.08/object (vendor sets the rate within AWS bounds)
Amazon Mechanical Turk: ~$0.02–$0.12/object depending on task complexity (separate MTurk pricing applies)

Free tier: 500 human reviews (42/month) for the first 12 months.

The Bedrock Evaluations native human task fee ($0.21/task) is higher than raw A2I because it bundles the worker-routing, UI rendering, and result aggregation managed by AWS. For high-volume human annotation budgets, direct A2I integration is cheaper. For occasional human spot-checks integrated into an eval workflow, the Bedrock native experience is worth the premium.

5.2 A2I Cost Budgeting

For a human review layer that samples 5% of production outputs for quality control:

At 100,000 production calls/day → 5,000 human reviews/day
At $0.04/object (marketplace worker): $200/day, $6,000/month
At $0.21/task (Bedrock native): $1,050/day — not viable at this scale

This delta makes A2I direct integration mandatory above ~$500/month in projected human review costs.

5.3 A2I vs Bedrock Evaluations Human Review — CUR Separation

A2I charges (AmazonA2I product code) and Bedrock inference charges (AmazonBedrock) appear as separate line items with separate product codes. Bedrock native human evaluation tasks appear within AmazonBedrock with HumanEvaluation as the operation. This means:

Mixed workflows (Bedrock eval job that routes flagged items to A2I) produce three separate cost buckets in CUR: Bedrock inference, Bedrock human task fees, and A2I object fees
Accurate human-review cost reporting requires a CUR query that unions all three

CUR SQL pattern for total human review cost (Athena):

SELECT
  bill_billing_period_start_date,
  SUM(line_item_unblended_cost) AS total_human_review_cost
FROM cur_data
WHERE (
  (line_item_product_code = 'AmazonBedrock'
    AND line_item_operation = 'HumanEvaluation')
  OR
  (line_item_product_code = 'AmazonA2I')
)
AND resource_tags_user_environment = 'eval'
GROUP BY 1
ORDER BY 1;

6. Eval Pipeline Architecture

6.1 Recommended Nightly Eval Run Architecture

EventBridge (cron: 02:00 UTC)
  → Lambda Orchestrator
      → Submit Batch Inference Job (model under eval) to Bedrock
      → Poll until complete (Step Functions wait state)
      → Submit Batch Inference Job (judge model) against outputs
      → Poll until complete
      → Lambda Scorer: aggregate scores, write to DynamoDB
      → CloudWatch custom metric: cost_per_eval_run, score_per_suite
      → SNS alert if cost deviation > 15% from 7-day average

Key design decisions:

Batch for both eval model and judge model: 50% cost reduction vs on-demand, acceptable for nightly cadence
Two-phase batch jobs: submit all model responses first, then fan out to judge. Avoids interleaving costs and enables caching the judge rubric across the entire batch
DynamoDB for score history: enables regression trend queries without S3 scan costs
Step Functions for wait states: avoids Lambda timeout issues on large batches (Lambda max 15 min; Bedrock batch jobs can run up to 24 hours)

6.2 CI/CD Integration Pattern

For pre-deployment eval gates:

# .github/workflows/eval-gate.yml (conceptual)
on:
  pull_request:
    paths: ['prompts/**', 'src/chains/**']

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - name: Submit Bedrock eval job
        run: |
          JOB_ID=$(aws bedrock create-evaluation-job \
            --job-name "ci-eval-${{ github.sha }}" \
            --evaluation-config '{"automated": {"datasetMetricConfigs": [...]}}' \
            --inference-config '{"models": [{"bedrockModel": {"modelIdentifier": "$EVAL_AIP_ARN"}}]}' \
            --output-data-config '{"s3Uri": "s3://eval-results/ci/${{ github.sha }}/"}' \
            --query jobArn --output text)
          
          # Poll with exponential backoff; fail PR if score < threshold
          python3 scripts/poll_eval_job.py --job-arn $JOB_ID --threshold 0.85

Using the eval-specific AIP ARN ($EVAL_AIP_ARN) ensures CI eval costs land in the eval cost bucket in CUR and never contaminate production spend attribution.

6.3 Cost-per-Eval-Run Tracking

Push a custom CloudWatch metric after each run:

cloudwatch.put_metric_data(
    Namespace='EvalPipeline',
    MetricData=[
        {
            'MetricName': 'CostPerRun',
            'Dimensions': [
                {'Name': 'EvalSuite', 'Value': suite_name},
                {'Name': 'Environment', 'Value': 'nightly'},
            ],
            'Value': total_cost_usd,
            'Unit': 'None',
        },
        {
            'MetricName': 'SamplesEvaluated',
            'Dimensions': [{'Name': 'EvalSuite', 'Value': suite_name}],
            'Value': sample_count,
            'Unit': 'Count',
        },
    ]
)

Derive cost-per-sample from these two metrics. CloudWatch Anomaly Detection on CostPerRun catches prompt drift or model-version changes that inflate eval costs within 1–2 standard deviations of the rolling baseline.

7. Batch API for Evaluations

7.1 The 50% Discount

Bedrock Batch Inference applies a flat 50% discount on on-demand rates for supported models including Anthropic Claude (all tiers), Meta Llama, Mistral, and Amazon Nova. The tradeoff is asynchrony: submit a JSONL file to S3, results return to S3 within 24 hours. Nightly eval runs are an ideal match — the result deadline is before the morning review, not before the next API call.

Batch input: JSONL file where each line is a Converse-API-compatible request object. Batch output: JSONL file mirroring the input structure with modelOutput appended per record.

7.2 Judge Model Batching

The two-phase batch pattern (eval model batch first, judge model batch second) captures the 50% discount on both legs:

Phase	Model	On-demand cost	Batch cost
Response generation	Claude Sonnet 4.6	$9.00/1K samples	$4.50/1K samples
Judge scoring	Claude Haiku 4.5	$2.20/1K samples	$1.10/1K samples
Combined		$11.20/1K samples	$5.60/1K samples

For a 5,000-sample nightly eval: on-demand $56 → batch $28. Over 365 nights: $10,220 → $5,110 annually.

7.3 Batch Limitations for Eval Pipelines

Minimum latency: ~15–30 minutes even for small jobs (Bedrock queues batch jobs internally)
No streaming: entire output available only on job completion
No prompt caching in batch mode: cache control headers are not honored in batch API requests
Maximum request file size: 512 MB per job; split large eval datasets into multiple jobs
Supported regions: not all regions support batch inference for all models — verify before architecture commits

Optimization choice: if the rubric caching savings exceed the batch discount savings for a given run size, prefer the on-demand path with caching. For a 1,000-sample run with an 800-token rubric and Sonnet judge:

Batch (no cache): saves $4.65 (50% of $9.30)
On-demand + cache: saves ~$8.15 (rubric savings $2.16 + output unchanged)
Cache beats batch for rubric-heavy eval runs under ~2,000 samples with Sonnet

Above ~2,000 samples, the batch discount begins to dominate even against cached rubric savings. Above 5,000 samples, run both legs in batch and accept the cache miss.

8. Model Selection for Judge Models

8.1 Cost/Accuracy Matrix (2026 Pricing)

Judge Model	Input $/M	Output $/M	Relative Accuracy	Best Use
Claude Haiku 4.5	$1.00	$5.00	High on binary/categorical criteria	Regression tests, pass/fail gates
Claude Sonnet 4.6	$3.00	$15.00	Strong across rubric complexity	Default judge for most eval suites
Claude Opus 4.6	$5.00	$25.00	Marginal gain over Sonnet on complex rubrics	Edge-case arbitration only

Haiku is 3× cheaper than Sonnet on input tokens and 3× cheaper on output. For simple rubrics (binary correctness, tone classification, hallucination detection on factual claims), Haiku-as-judge produces agreement rates with human reviewers comparable to Sonnet. The degradation appears on nuanced rubrics: multi-dimensional quality scoring, logical consistency across long completions, and cross-document reasoning tasks.

8.2 When Judge Degradation Matters

Haiku judge degradation is material when:

The rubric has more than 3 simultaneous dimensions being scored
Completions exceed 1,500 tokens (Haiku’s attention to detail degrades on longer completions)
The evaluation is a production release gate where a false-positive pass has business consequences
You are measuring subtle quality properties (tone appropriateness, cultural sensitivity)

Haiku degradation is acceptable for:

Nightly regression tests catching regressions > 5% from baseline
High-volume cost-of-failure-is-low screening (surface candidates for human review)
Structural correctness checks (JSON validity, schema adherence, required fields)
Continuous monitoring of production samples where statistical aggregation smooths individual judge errors

8.3 Tiered Judge Routing

A cost-optimal pattern routes judge calls by rubric complexity:

def select_judge(rubric_complexity: str, sample_count: int) -> str:
    if rubric_complexity == "binary":
        return HAIKU_AIP_ARN          # $1.00/$5.00 — cheapest
    elif rubric_complexity == "multi-dim" and sample_count < 200:
        return SONNET_AIP_ARN         # $3.00/$15.00 — default
    elif rubric_complexity == "multi-dim" and sample_count >= 200:
        return HAIKU_AIP_ARN          # Accept statistical smoothing at scale
    else:  # "complex" rubrics, arbitration
        return OPUS_AIP_ARN           # $5.00/$25.00 — reserved for edge cases

At a representative production eval suite (70% binary checks, 25% multi-dim, 5% complex arbitration), blended judge cost ≈ $1.50/1,000 samples vs $3.00 all-Sonnet — a 50% reduction with no material accuracy loss on the binary majority.

9. Eval Dataset Storage Costs

9.1 S3 Storage Tiers for Eval Datasets

Dataset Type	Recommended Tier	Cost/GB-month	Rationale
Active eval sets (accessed weekly)	S3 Standard	$0.023	Low enough to not optimize
Historical eval outputs (audit trail)	S3 Standard-IA	$0.0125	Accessed <1/month per run
Archived baselines (>90 days old)	S3 Glacier Instant	$0.004	Needed for regression forensics

A lifecycle policy on the eval output bucket:

{
  "Rules": [{
    "Filter": {"Prefix": "eval-outputs/"},
    "Status": "Enabled",
    "Transitions": [
      {"Days": 30, "StorageClass": "STANDARD_IA"},
      {"Days": 90, "StorageClass": "GLACIER_IR"}
    ],
    "Expiration": {"Days": 365}
  }]
}

This lifecycle on a bucket accumulating 1 GB/day of eval output: Year-1 storage cost ≈ $3.20/month (blended across tiers) vs $8.40/month on Standard throughout — 62% storage savings.

9.2 Data Transfer Costs

Bedrock reads eval input from S3 in the same region — no data transfer charges. Eval output written to S3 by Bedrock is also free (PUT is charged at S3 rate: $0.005/1,000 requests, negligible). Cross-region S3 reads for eval datasets that live in a different region than the Bedrock endpoint: $0.02/GB — avoid by co-locating datasets with the Bedrock region.

9.3 Bedrock Knowledge Base Eval Costs

When evaluating RAG systems backed by a Bedrock Knowledge Base:

Vector index storage: $0.10/GB/month (OpenSearch Serverless under the hood)
Retrieval requests: ~$0.10/1,000 queries (service-dependent, varies by KB type)
Embedding model calls: billed as standard inference on the embedding model

For a 1 GB KB with 1,000 retrieval calls per eval run, monthly overhead is ~$0.10 storage + $0.10 retrieval = $0.20/eval run. Negligible unless running multiple KB evals per day.

10. Regression Detection — Tracking Cost-per-Eval Over Time

10.1 The Signal: Cost Inflation as a Proxy for Drift

Cost-per-eval-sample is a leading indicator of two failure modes:

Prompt drift: someone modified the eval system prompt or rubric, increasing token counts
Model version changes: a provider updated the underlying model weights, causing longer or shorter completions (Bedrock on-demand models can change versions without explicit notice on non-pinned model IDs)

A sustained 10% increase in cost-per-sample with no corresponding score change is a flag for token bloat. A spike in cost without a score change, followed by a return to baseline, suggests a transient model behavior shift.

10.2 Cost-per-Eval CloudWatch Alarm

cloudwatch.put_metric_alarm(
    AlarmName='EvalCostPerSampleAnomaly',
    AlarmDescription='Eval cost per sample exceeds 2 std devs from 14-day baseline',
    Namespace='EvalPipeline',
    MetricName='CostPerSample',
    Dimensions=[{'Name': 'EvalSuite', 'Value': suite_name}],
    Period=86400,  # daily
    EvaluationPeriods=1,
    ComparisonOperator='GreaterThanUpperThreshold',
    ThresholdMetricId='anomaly_detection_band',
    Metrics=[
        {'Id': 'm1', 'MetricStat': {'Metric': {...}, 'Period': 86400, 'Stat': 'Average'}},
        {'Id': 'anomaly_detection_band', 'Expression': 'ANOMALY_DETECTION_BAND(m1, 2)'}
    ],
    TreatMissingData='ignore',
)

10.3 Baseline the Eval Cost at Launch

At every new eval suite launch:

Record baseline_cost_per_sample, baseline_input_tokens_per_sample, baseline_output_tokens_per_sample to Parameter Store
Tag the Parameter Store entry with eval-suite-id and launched-date
On each nightly run, compare actuals to baseline and emit a deviation percentage metric
Set threshold: >15% deviation triggers investigation; >30% deviation blocks deployment gates if eval is in CI

11. FinOps Tagging for Eval Infrastructure

11.1 Application Inference Profiles (AIPs) as the Foundation

AIPs are the primary Bedrock mechanism for separating eval from production costs in CUR. Create a dedicated AIP for each eval suite:

bedrock.create_inference_profile(
    inferenceProfileName='nightly-eval-judge-sonnet',
    description='Judge model for nightly regression eval suite',
    modelSource={'copyFrom': 'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-6-v1:0'},
    tags=[
        {'key': 'environment', 'value': 'eval'},
        {'key': 'workload', 'value': 'nightly-eval'},
        {'key': 'eval-suite', 'value': 'regression-v2'},
        {'key': 'cost-center', 'value': 'ml-platform'},
        {'key': 'owner', 'value': 'platform-eng'},
    ]
)

All inference through this AIP flows to CUR tagged with these values. In Cost Explorer, filter resource_tags_user_environment = eval to isolate all eval spend from production.

11.2 Mandatory Tag Set for Eval Infrastructure

Tag Key	Values	Applied To
`environment`	`eval`, `prod`, `dev`	AIP, S3 bucket, Lambda, Step Functions
`workload`	`nightly-eval`, `ci-eval`, `human-review`	AIP, Lambda
`eval-suite`	`regression-v2`, `rag-quality`, `safety-screen`	AIP, S3 prefix
`cost-center`	`ml-platform`, `product-{team}`	All resources
`owner`	team slug	All resources

Tags on the AIP flow to CUR automatically. Tags on S3 buckets and Lambda functions require Cost Allocation Tag activation in the Billing console — takes up to 24 hours to appear in CUR and are not retroactive.

11.3 IAM Principal-Based Cost Attribution (April 2026)

Bedrock now supports cost allocation by IAM principal in CUR 2.0 and Cost Explorer. If your eval pipeline runs under a dedicated IAM role (arn:aws:iam::123456789:role/eval-pipeline-role), all inference through that role is attributable without needing AIPs.

CUR 2.0 column: identity_user_arn — contains the IAM principal that made each request. This is available at no additional cost and requires no AIP setup. It is the lowest-friction path for teams that have not yet adopted AIPs.

Limitation: IAM-based attribution is read-only — you cannot apply cost allocation tags via IAM identity alone. AIPs remain necessary for multi-dimensional tagging (cost-center + environment + eval-suite simultaneously).

11.4 CUR Query: Monthly Eval Spend by Suite

-- Athena query for CUR 2.0
SELECT
  resource_tags_user_eval_suite AS eval_suite,
  resource_tags_user_environment AS environment,
  SUM(line_item_unblended_cost) AS total_cost_usd,
  SUM(CASE WHEN line_item_usage_type LIKE '%-Input-Tokens' THEN line_item_usage_amount ELSE 0 END) AS input_tokens,
  SUM(CASE WHEN line_item_usage_type LIKE '%-Output-Tokens' THEN line_item_usage_amount ELSE 0 END) AS output_tokens
FROM cur_2_0_table
WHERE
  line_item_product_code = 'AmazonBedrock'
  AND resource_tags_user_environment = 'eval'
  AND bill_billing_period_start_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY 1, 2
ORDER BY total_cost_usd DESC;

12. Budget Controls and Guardrails

12.1 AWS Budgets for Eval Workloads

Create a filtered Budget scoped to eval tags:

budgets.create_budget(
    AccountId=account_id,
    Budget={
        'BudgetName': 'bedrock-eval-monthly',
        'BudgetLimit': {'Amount': '500', 'Unit': 'USD'},
        'BudgetType': 'COST',
        'TimeUnit': 'MONTHLY',
        'CostFilters': {
            'TagKeyValue': ['user:environment$eval'],
            'Service': ['Amazon Bedrock'],
        },
    },
    NotificationsWithSubscribers=[{
        'Notification': {
            'NotificationType': 'ACTUAL',
            'ComparisonOperator': 'GREATER_THAN',
            'Threshold': 80.0,
            'ThresholdType': 'PERCENTAGE',
        },
        'Subscribers': [{'SubscriptionType': 'EMAIL', 'Address': 'ml-platform@company.com'}]
    }]
)

12.2 SCP to Prevent Untagged Eval Jobs

A Service Control Policy can block Bedrock CreateEvaluationJob calls that lack the required environment tag:

{
  "Effect": "Deny",
  "Action": "bedrock:CreateEvaluationJob",
  "Resource": "*",
  "Condition": {
    "Null": {"aws:RequestTag/environment": "true"}
  }
}

This is a forcing function — no eval job can be submitted without a tag, preventing invisible cost accumulation.

Key Numbers Reference

Item	Price
Bedrock algorithmic eval scoring	$0 (inference only)
Bedrock human evaluation task fee	$0.21/task
A2I custom workflow (private workforce)	$0.02–$0.04/object
A2I custom workflow (marketplace)	$0.04–$0.08/object
Batch inference discount	50% off on-demand
Prompt cache write surcharge	1.25× input price
Prompt cache read discount	0.10× input price (90% off)
Claude Haiku 4.5	$1.00/$5.00 per M input/output
Claude Sonnet 4.6	$3.00/$15.00 per M input/output
Claude Opus 4.6	$5.00/$25.00 per M input/output
S3 Standard	$0.023/GB-month
S3 Standard-IA	$0.0125/GB-month
S3 Glacier Instant Retrieval	$0.004/GB-month
Bedrock KB vector storage	~$0.10/GB-month
Bedrock KB retrieval	~$0.10/1,000 queries