← Agent Frameworks 🕐 15 min read

Agent Frameworks

OpenCost & Kubecost for AI/LLM Workload Cost Observability on Kubernetes

Executive Answer · March 2026

Neither OpenCost nor Kubecost tracks Bedrock API token costs natively. The correct architecture for an enterprise running LiteLLM on EKS routing to Bedrock is a **three-layer cost stack**:

Platform engineering synthesis — June 2026

Neither OpenCost nor Kubecost tracks Bedrock API token costs natively. The correct architecture for an enterprise running LiteLLM on EKS routing to Bedrock is a three-layer cost stack:

LiteLLM proxy — per-request token cost, tagged by team/model/key (Prometheus or PostgreSQL)
OpenCost or Kubecost — LiteLLM pod infrastructure cost (CPU, memory, GPU if local inference)
AWS Bedrock CUR 2.0 + Application Inference Profiles — Bedrock API spend, attributed by workload tag

These three layers are joined at the team or application grain — not at the individual request level. The join key is a shared label/tag (e.g. app=litellm-prod, team=platform) applied consistently across K8s pod labels, LiteLLM x-litellm-tag metadata, and Bedrock Application Inference Profile tags.

Tool choice summary:

OpenCost (free, CNCF) is sufficient for namespace/label cost allocation with GPU support from v1.111+. Use it when you have engineering bandwidth and want zero licensing cost.
Kubecost Enterprise is required for: multi-cluster federation, CUR reconciliation (spot/EDP pricing accuracy), Bedrock-side AWS service costs alongside K8s costs, and non-engineering stakeholder reporting. GPU cost via DCGM is available in all tiers including free from v2.4.
A Bedrock plugin for OpenCost does not exist today. An OpenAI plugin ships in the opencost/opencost-plugins repo. Building a Bedrock plugin is feasible using the FOCUS CustomCost interface — see the implementation path below.

Research Map

Seed question: Can OpenCost or Kubecost track the full cost of an enterprise AI gateway (LiteLLM on EKS → AWS Bedrock)?

Adjacent systems discovered:

Bedrock Application Inference Profiles (CUR-visible workload tags, analogous to K8s namespaces)
Bedrock model invocation logs (per-request token counts; CUR cannot carry per-requestId)
AWS EKS Split Cost Allocation Data (Oct 2025: supports up to 50 K8s labels per pod as CUR cost allocation tags; Sep 2025: adds NVIDIA/AMD GPU, Trainium, Inferentia)
LiteLLM Prometheus integration (per-request cost metric litellm_spend_metric with team/model/key labels)
NVIDIA DCGM Exporter → Prometheus → Kubecost/OpenCost GPU cost attribution pipeline
OpenCost plugin framework v0.0.16 (June 2026) with FOCUS CustomCost interface

Assumptions revised:

Assumption “CUR 2.0 carries per-request Bedrock costs” is wrong. CUR aggregates by usage type + operation + pricing/resource over an hour or a day. No requestId in CUR. Per-request attribution requires joining Bedrock model invocation logs with CUR at the model+usage-type grain.
Assumption “OpenCost has a Bedrock plugin” is wrong. An OpenAI plugin exists; no Bedrock plugin as of June 2026.
Assumption “Kubecost natively surfaces Bedrock spend” is wrong. Kubecost tracks K8s + AWS EC2/RDS/S3 via CUR, but Bedrock spend requires separate tooling.

Cross-Phase Map

Phase	What Happens	Tool / Contract
Native capture	LiteLLM logs token counts + calculated cost per request to PostgreSQL `LiteLLM_SpendLogs`	LiteLLM OSS, no config needed
Configured telemetry	`PROMETHEUS_MULTIPROC_DIR` + `/metrics` endpoint enables per-model/team/key cost counters	`litellm_spend_metric{model, team, hashed_api_key}`
K8s infra cost	OpenCost/Kubecost scrape node cost; allocate to LiteLLM pod namespace via CPU+memory requests	OpenCost ≥1.110, Kubecost ≥2.0
GPU node cost	DCGM Exporter → Prometheus → Kubecost v2.4+ / OpenCost v1.111+ attribute GPU cost to pod/namespace	Requires NVIDIA GPU Operator + DCGM Exporter installed
Bedrock API cost	CUR 2.0 line items by model + token type + service tier + Application Inference Profile tag	AWS Billing; activate tags in console; 24h lag
Cross-layer join	Join LiteLLM `team` label + K8s namespace + Bedrock profile tag in Athena/QuickSight	Shared `team=` key; no native join tool exists
Cost attribution	Cost per request = (LiteLLM pod CPU+mem cost / total requests) + (Bedrock token cost from CUR)	Computed in Athena or custom dashboard
Budget enforcement	LiteLLM enforces per-key/per-team spend limits; no Bedrock-side enforcement in K8s tooling	LiteLLM `max_budget` + `budget_duration`

Section 1: OpenCost AI/LLM Support

What OpenCost tracks natively

OpenCost tracks Kubernetes infrastructure costs only: CPU, memory, storage, and network for pods/namespaces/deployments/labels. It does not natively track any API call costs (Bedrock, OpenAI, Anthropic, etc.).

For cloud infrastructure, OpenCost fetches on-demand pricing from the AWS Price List API and optionally reconciles via CUR (enterprise pattern; the core project uses billing APIs for “ballpark” accuracy).

Plugin system (v1.110.0+, plugin repo v0.0.16)

The opencost/opencost-plugins repo implements a FOCUS-compliant CustomCost interface. Available plugins as of June 2026:

Datadog — reference implementation / monitoring costs
OpenAI — OpenAI API spend (operational)
MongoDB Atlas — database costs

No Bedrock plugin exists. The OpenAI plugin is the closest analog.

How the plugin interface works

OpenCost core → gRPC query to plugin binary →
  Plugin calls upstream API (e.g. OpenAI Usage API) →
  Plugin returns FOCUS CustomCostResponse list →
  OpenCost stores in-memory (re-queried on pod restart)

FOCUS CustomCost fields returned by a plugin:

zone, account_name, charge_category, description,
resource_name, resource_type, provider_id,
billedCost, listCost, list_unit_price,
usage_quantity, usage_unit, domain, cost_source

CustomCostExtendedAttributes is optional and carries arbitrary key-value metadata.

Plugin interface requirement (GetCustomCosts function):

Input: CustomCostRequest{Start, End, Resolution} (resolution = hour or day)
Output: []CustomCostResponse conforming to FOCUS spec

Deployment via Helm (init container pattern):

# values.yaml
opencost:
  plugins:
    enabled: true
    folder: /opt/opencost/plugin
    configs:
      openai: |
        {"openai_api_key": "sk-proj-XXXX"}
  extraEnv:
    - name: PLUGIN_EXECUTABLE_DIR
      value: /opt/opencost/plugin/bin
    - name: PLUGIN_CONFIG_DIR
      value: /opt/opencost/plugin/config
    - name: CUSTOM_COST_ENABLED
      value: "true"

Building a Bedrock plugin (implementation path)

A Bedrock CustomCost plugin would:

Call aws bedrock Cost Explorer or query the CUR S3 bucket via Athena SDK
Return per-model, per-usage-type aggregates in the FOCUS CustomCost schema
Map: resource_name=model-id, usage_unit=tokens, billedCost=USD, charge_category=Usage

The FOCUS spec does not have a token_count field — you would carry token counts in CustomCostExtendedAttributes. The usage_quantity field carries a numeric value; use token count as the unit with usage_unit="1K input tokens" etc.

Reference: opencost/opencost-plugins GitHub — Datadog plugin is the reference implementation.

2026 roadmap (source: CNCF blog Jan 2026)

“KubeModel” — Data Model 2.0 for Kubernetes complexity (mentee Sparsh)
“AI costing features to track AI usage” — explicit roadmap item, no shipped artifact yet
Supply chain security
MCP server for AI agent natural-language queries of cost data (shipped 2025)

Contract level: roadmap items = community pattern / inference; not shipped features.

Section 2: Kubecost AI Features

GPU cost monitoring (v2.4, released Oct 22 2024)

Kubecost v2.4 is the first release with NVIDIA GPU cost attribution. Available in all tiers including free.

Requirements:

NVIDIA GPU Operator installed on cluster
DCGM Exporter installed on GPU nodes, exporting to Prometheus

DCGM Exporter install:

helm upgrade -i dcgm dcgm-exporter \
  --repo https://nvidia.github.io/dcgm-exporter/helm-charts \
  -n dcgm-exporter --create-namespace \
  -f values-dcgm.yaml

values-dcgm.yaml must include node affinity targeting GPU node labels.

Metrics surfaced by Kubecost v2.4:

GPU utilization (actual usage % per container via DCGM_FI_DEV_GPU_UTIL)
GPU workload idle (requested but unused GPU resources)
Infrastructure idle (GPUs not requested at all)
GPU efficiency ratio (actual vs requested)

How GPU cost is attributed: Starting v2.4, allocated GPU cost is determined by actual container utilization (via DCGM), not by GPU presence alone.

Known hard limitation: NVIDIA does not support DCGM utilization metrics for containers using GPU sharing (MIG, time-slicing). Kubecost will display GPU cost of zero for those workloads.

LLM / Bedrock cost tracking

Kubecost does not track Bedrock API spend. It tracks K8s workload costs and, in Enterprise, AWS services via CUR (EC2, RDS, S3) — but Bedrock is not a surfaced AWS service integration in Kubecost as of June 2026.

The gap: Kubecost sees the compute cost of running LiteLLM pods. It does not see the Bedrock API token costs those pods generate.

AWS integration depth

Capability	OpenCost (free)	Kubecost Free	Kubecost Enterprise
K8s namespace/pod/label cost	Yes	Yes	Yes
GPU cost (DCGM)	v1.111+	v2.4+	v2.4+
CUR reconciliation (spot/EDP)	No	No	Yes
AWS Split Cost Allocation (K8s labels in CUR)	Works via AWS native	Works via AWS native	Works via AWS native
Multi-cluster federated view	Manual	No	Yes
Bedrock spend	No	No	No
AWS RDS/S3/EC2 alongside K8s	No	No	Yes
Rightsizing recommendations	Basic	Basic	Full

Kubecost Enterprise pricing (2026): ~$3.42/container-hour (AWS Marketplace); or tiered per-node pricing for larger deployments.

Section 3: LiteLLM Infrastructure Cost on Kubernetes

LiteLLM pod cost with OpenCost/Kubecost

Both tools track LiteLLM pods the same as any other workload. Apply a label to LiteLLM pods:

# LiteLLM deployment labels
labels:
  app: litellm-proxy
  team: platform
  cost-center: ai-gateway

OpenCost/Kubecost then surfaces cost at the namespace and label grain:

CPU cost per pod (proportional to requests/limits)
Memory cost per pod
Network egress cost (if configured)

Namespace-level cost allocation

# OpenCost API query for namespace cost
curl "http://opencost:9003/allocation/compute?window=7d&aggregate=namespace" | jq .

For LiteLLM, the interesting namespace is wherever it runs (ai-platform, llm-gateway, etc.).

Cost per request calculation

OpenCost/Kubecost give you $/day for the LiteLLM namespace. LiteLLM’s Prometheus endpoint gives you request count. Divide:

infra_cost_per_request = namespace_daily_cost_usd / daily_request_count

Total cost per request:

total_cost_per_request = infra_cost_per_request
                        + litellm_spend_metric (Bedrock token cost, in USD, per request)

LiteLLM Prometheus metrics for this calculation (all OSS, no enterprise required):

Metric	Labels	Use
`litellm_spend_metric`	`model, team, hashed_api_key, end_user`	Token cost in USD (from LiteLLM’s internal cost map)
`litellm_total_tokens_metric`	`model, team, requested_model`	Token throughput
`litellm_input_tokens_metric`	same	Input tokens
`litellm_output_tokens_metric`	same	Output tokens
`litellm_input_cached_tokens_metric`	same	Cache-read tokens (significantly cheaper in Bedrock)

Critical bug (open as of v1.80.5): LiteLLM GitHub issue #17415 — Prometheus metrics do not update for Bedrock requests when litellm_params is None. The litellm_proxy_total_requests_metric_total counter stays static for Bedrock-routed requests while working for Vertex AI. A one-line patch is proposed: change litellm_params.get("metadata", {}) to (litellm_params or {}).get("metadata", {}) in litellm/integrations/prometheus.py line ~995. Verify this is patched before relying on Bedrock Prometheus metrics in production.

AWS Split Cost Allocation for EKS (Oct 2025)

AWS natively supports up to 50 Kubernetes pod labels as CUR cost allocation tags via Split Cost Allocation Data. This means LiteLLM pod labels like team=platform appear as CUR columns, allowing EC2 compute cost to be attributed at the pod/namespace level without any third-party tooling.

Setup:

Enable Split Cost Allocation Data in AWS Billing console (management account)
Label LiteLLM pods with desired cost allocation labels
Activate tags in Billing console (24h propagation)
Query CUR via Athena with resourceTags/team = 'platform' filter

Sep 2025 addition: EKS Split Cost Allocation now supports NVIDIA/AMD GPU, Trainium, and Inferentia EC2 instances — meaning GPU node costs are also attributed at the pod level in CUR.

Section 4: GPU Workload Cost Tracking

Architecture: vLLM or Ollama alongside Bedrock API calls

When an enterprise runs both local GPU inference (vLLM, Ollama) and Bedrock API routing through LiteLLM, the cost picture spans two cost types:

LiteLLM pod
├── Route A: local vLLM/Ollama (GPU inference pod, same or different namespace)
│   └── Cost = GPU node cost / utilization, attributed via DCGM
└── Route B: Bedrock API
    └── Cost = Bedrock token cost via CUR + Application Inference Profile

OpenCost GPU support

OpenCost added GPU cost attribution in v1.111. It uses DCGM metrics from Prometheus to attribute GPU node costs to workloads. Configuration requires:

NVIDIA GPU Operator running (provides DCGM Exporter as a DaemonSet)
Prometheus scraping DCGM Exporter at port 9400 (/metrics)
OpenCost configured to read from the same Prometheus

OpenCost vs Kubecost on GPU: Both use the same underlying data source (DCGM → Prometheus). Kubecost’s v2.4 GPU report adds a polished UI with efficiency widgets and idle cost breakdowns. OpenCost requires more manual Grafana dashboard configuration for equivalent visibility.

DCGM metric stack

Key DCGM metrics used for cost attribution:

Metric	Meaning	Used by
`DCGM_FI_DEV_GPU_UTIL`	GPU utilization %	Both tools
`DCGM_FI_DEV_FB_USED`	GPU memory used	Both tools
`DCGM_FI_DEV_FB_FREE`	GPU memory free	Both tools
`DCGM_FI_PROF_GR_ENGINE_ACTIVE`	Graphics engine active fraction	Kubecost v2.4 idle detection

Alert pattern for idle GPU detection:

# Prometheus alert rule
- alert: GPUUnderutilized
  expr: DCGM_FI_DEV_GPU_UTIL < 10
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "GPU idle >15min in {{ $labels.namespace }}"

Neither tool supports GPU cost attribution for containers using NVIDIA MIG or time-slicing. When a GPU is shared across pods, DCGM does not expose per-process utilization and both tools show GPU cost as zero for those workloads. This is an NVIDIA DCGM limitation, not a Kubecost/OpenCost limitation.

vLLM cost per token (practical numbers)

Self-hosted vLLM on a single H100 SXM5 (spot or reserved):

H100 cloud (AWS p4de equivalent): ~$2.50–$4.10/hr on competitive providers, $6.88/hr AWS on-demand
Typical inference throughput for a 70B model: ~4,000–8,000 tokens/sec

At $3.00/hr and 6,000 tokens/sec sustained:

cost_per_million_tokens = (3.00 / 3600) / 6000 * 1,000,000 = ~$0.14/M tokens

Bedrock Claude Sonnet 4.6 input: comparable range. Output tokens at ~5x input price change the comparison materially for output-heavy workloads.

Section 5: FinOps for AI Platform Decision

GPU on-prem vs Bedrock API break-even

The structural reality (2026): GPU cloud pricing has collapsed. H100 spot/reserved on competitive providers is $1.03–$2.50/hr. H100 SXM5 8-card nodes are ~$25,000–$30,000 per GPU purchase price.

At $2.50/hr rental, break-even on a purchased H100 = ~10,000–12,000 hours = 14–16 months at 100% utilization. Production LLM inference teams typically operate at 40–65% GPU utilization due to traffic variability and batching limits, shifting the real break-even to 24–36 months or never.

The Bedrock comparison is model-specific. For Claude Sonnet 4.6 on Bedrock:

Input tokens: standard tier pricing (see CUR usage type USE1-Claude4.6Sonnet-input-tokens)
Output tokens: ~5x input price
Cache read tokens: significantly cheaper than input (prompt caching)

For a workload generating 1B tokens/month with 50/50 input/output mix, compute the Bedrock monthly cost and compare to a reserved H100 instance. The break-even shifts as: (a) output-heavy workloads favor on-prem; (b) bursty/variable workloads favor Bedrock; © compliance requirements forcing on-prem shift the calculus independent of cost.

Practical guidance: At under 70% sustained GPU utilization, Bedrock (or any API provider) wins on pure cost. Above 70% sustained utilization with predictable load — on-prem wins on a 3-year horizon only if priced against AWS on-demand. Against competitive GPU clouds ($2.50/hr H100), on-prem rarely wins on cost alone at current GPU prices.

Unified cost view architecture

The only way to get a true unified cost view across LiteLLM infra + Bedrock API + local GPU is to build it in Athena or a cost aggregation platform. No single off-the-shelf tool does this as of June 2026.

Data sources:
├── OpenCost/Kubecost Prometheus metrics → LiteLLM pod $/day
├── LiteLLM PostgreSQL SpendLogs → per-request token cost + team/model breakdown
├── Bedrock CUR 2.0 (S3 → Athena) → hourly token cost by model + Application Inference Profile
└── EKS Split Cost Allocation CUR → pod-level EC2 cost (including GPU nodes)

Join grain:
  K8s label `team=X`
  = LiteLLM `x-litellm-tag: team:X`
  = Bedrock Application Inference Profile tag `team=X`

Unified view table (Athena):
  team | date | k8s_infra_cost | bedrock_token_cost | gpu_node_cost | total_cost | request_count | cost_per_request

Bedrock Application Inference Profiles are the critical enabler. They are logical groupings (think K8s namespaces, but for Bedrock) that appear as CUR tags. LiteLLM can be configured to route to a specific Application Inference Profile ARN per team/route, causing Bedrock spend to appear segmented by workload in CUR.

# LiteLLM config: route team A to Bedrock profile
model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/arn:aws:bedrock:us-east-1:123456789:inference-profile/team-a-profile
      aws_region_name: us-east-1

Finout is the closest commercial platform that unifies OpenAI/Anthropic/Bedrock/Vertex API costs alongside Kubernetes spend in a single “MegaBill.” Vantage and CloudZero offer similar but more cloud-focused views. None of these replace the need for LiteLLM-level per-request attribution.

Format/Tool Matrix

Capability	OpenCost OSS	Kubecost Free	Kubecost Enterprise	LiteLLM OSS	AWS Native (CUR+Split)
K8s pod/namespace cost	Yes	Yes	Yes	No	Partial (Split Cost Alloc)
GPU cost (DCGM)	v1.111+	v2.4+	v2.4+	No	v Sep 2025+
Bedrock API cost	Plugin (build it)	No	No	Per-request	Yes (CUR 2.0)
OpenAI API cost	Plugin (shipped)	No	No	Per-request	No
Per-request token cost	No	No	No	Yes	No (hourly aggregate)
Per-team cost	Via labels	Via labels	Via labels	Yes (native)	Via K8s labels
CUR reconciliation	No	No	Yes	No	Native
Multi-cluster	Manual	No	Yes	N/A	Yes
Budget enforcement	No	No	No	Yes (per-key/team)	No
License cost	Free	Free	~$3.42/container-hr	Free (Enterprise for Prometheus)	Included in AWS

Hard Edges

CUR has no per-requestId. CUR aggregates Bedrock cost by usage type + model + hour/day. To attribute cost to an individual LiteLLM request, you must join Bedrock model invocation logs (separate S3 destination) with CUR at the model+usage-type grain. This join does not exist in any off-the-shelf tool.
LiteLLM Prometheus bug on Bedrock. Issue #17415 (v1.80.5): litellm_proxy_total_requests_metric_total and related counters do not update for Bedrock requests. Patch: null-check litellm_params in prometheus.py:995. Verify fix is merged before relying on Prometheus metrics for Bedrock cost.
OpenCost plugins are in-memory. Plugin cost data is re-queried on every pod restart. For a Bedrock plugin pulling from CUR (which has 24h lag), this means startup cost is zero until the plugin re-fetches. Design for this: store an external cost cache (Redis or Postgres) outside the OpenCost pod.
GPU sharing zeroes out DCGM costs. MIG and time-slicing workloads show zero GPU cost in both Kubecost and OpenCost. If vLLM or Ollama uses GPU sharing, cost attribution is lost.
EKS Split Cost Allocation 50-label limit. Labels are alphabetically sorted; only the first 50 are imported. Ensure critical tags (team=, app=, cost-center=) sort to the front alphabetically or use a prefix convention (aaa-team=).
Bedrock Application Inference Profile tag activation lag. Tags must be activated in the AWS Billing console at the management account level. After creation, 24h for tags to appear; another 24h after activation for data to appear in CUR. Do not expect same-day visibility.
Cache token accounting. Bedrock CUR has four token types: input, output, cache-read, cache-write. Cache-read is cheaper; cache-write is more expensive than input. If you only sum input + output tokens, your cost analysis will be wrong for workloads using prompt caching heavily. LiteLLM tracks litellm_input_cached_tokens_metric separately — ensure dashboards include it.
OpenCost accuracy vs Kubecost on spot instances. OpenCost uses on-demand pricing from the AWS Price List API. If your LiteLLM or vLLM pods run on spot instances (common for GPU inference), OpenCost will overstate infrastructure cost significantly. Kubecost Enterprise CUR reconciliation corrects this. Alternative: use EKS Split Cost Allocation which pulls from CUR directly.

Verification Ledger

Claim	Source	Level
OpenCost plugin system, FOCUS interface, Datadog + OpenAI + MongoDB Atlas plugins	opencost/opencost-plugins GitHub; opencost.io/docs/integrations/plugins	Official docs verified
OpenCost 2026 roadmap: AI costing features planned, not shipped	CNCF blog Jan 2026	Official docs verified
Kubecost v2.4 GPU monitoring via DCGM, all tiers, Oct 2024	Apptio blog	Official docs verified
LiteLLM Prometheus metrics table (all OSS)	docs.litellm.ai/docs/proxy/prometheus	Official docs verified
LiteLLM Bedrock Prometheus bug #17415	BerriAI/litellm issue #17415	Source/code verified
Bedrock CUR 2.0 schema: four token types, no per-requestId	AWS Bedrock CUR docs	Official docs verified
EKS Split Cost Allocation K8s labels (Oct 2025, 50-label limit)	AWS docs	Official docs verified
EKS Split Cost Allocation GPU support (Sep 2025)	AWS what’s new	Official docs verified
Bedrock Application Inference Profiles appear in CUR	Finout blog	Community pattern
GPU utilization break-even: 70%+ for on-prem to win	Spheron on-prem vs cloud analysis	Community pattern
GPU sharing zeroes out DCGM metrics	Kubecost blog via Apptio	Official docs verified
LiteLLM requestMetadata for Bedrock multi-tenant attribution	AWS ML blog	Official docs verified

Visual Explainer: Unified Cost Stack

┌─────────────────────────────────────────────────────────────────────┐
│                     ENTERPRISE AI COST STACK                        │
├─────────────────────┬───────────────────────┬───────────────────────┤
│  Layer 1: K8s Infra │  Layer 2: Token Costs │  Layer 3: GPU Infra   │
│  (OpenCost/Kubecost)│  (LiteLLM + Bedrock)  │  (DCGM + OpenCost)   │
├─────────────────────┼───────────────────────┼───────────────────────┤
│ LiteLLM pods:       │ Per-request cost:      │ vLLM/Ollama pods:     │
│ CPU + memory $/day  │ litellm_spend_metric   │ GPU node $/hr via     │
│ attributed by       │ (Prometheus, OSS)      │ DCGM_FI_DEV_GPU_UTIL  │
│ namespace / label   │                        │ per namespace         │
│                     │ Bedrock hourly cost:   │                       │
│ Source: OpenCost    │ CUR 2.0 by model +     │ Source: Kubecost v2.4 │
│ allocation API      │ token type + profile   │ or OpenCost v1.111    │
│ or Kubecost UI      │                        │                       │
├─────────────────────┴───────────────────────┴───────────────────────┤
│                    JOIN GRAIN: team= label/tag                       │
│   K8s label team=X = LiteLLM tag team:X = Bedrock profile team-X   │
├─────────────────────────────────────────────────────────────────────┤
│              UNIFIED VIEW (Athena / QuickSight / Grafana)            │
│  team | infra_cost | bedrock_token_cost | gpu_cost | total | CPR    │
└─────────────────────────────────────────────────────────────────────┘

CPR = cost per request

Next Experiments

Build and test a Bedrock OpenCost plugin. Start from the Datadog reference implementation in opencost/opencost-plugins. The Bedrock plugin would call the AWS Cost Explorer API (GetCostAndUsage filtered to SERVICE = Amazon Bedrock) and return hourly aggregates per model as FOCUS CustomCost records. This closes the biggest gap in the current stack.
Verify LiteLLM Bedrock Prometheus bug fix. Pin a LiteLLM version post-v1.80.5 and send 100 Bedrock requests; check that litellm_spend_metric increments with model=bedrock/... labels. This converts the bug report from “closed” to locally verified.
Validate Bedrock Application Inference Profile CUR appearance. Create two profiles (team-a, team-b), route LiteLLM to each, run requests, wait 24h, query CUR in Athena. Verify that resourceTags/team column is populated and joinable with EKS Split Cost Allocation data on the same team value.
Measure OpenCost spot instance pricing error. Run OpenCost against an EKS cluster with a mix of spot and on-demand GPU nodes. Compare OpenCost’s cost estimate against CUR actuals. This quantifies whether Kubecost Enterprise CUR reconciliation is worth the cost for GPU-heavy workloads.
End-to-end cost per request measurement. Instrument the full stack: OpenCost namespace cost / LiteLLM request count + litellm_spend_metric. Run 10K requests against Bedrock and 10K against vLLM. Produce a cost-per-request comparison table for the same model family across both routes.