← Agent Frameworks 🕐 16 min read

Agent Frameworks

OpenCost and Kubecost for AI Workload Cost Allocation (2026)

1. OpenCost: CNCF Project Status, Spec Scope, and Gaps · March 2026

OpenCost is the CNCF incubating project that defines both an **open specification** and a **reference implementation** for Kubernetes cost allocation.

Platform engineering synthesis — June 2026

What OpenCost Is

OpenCost is the CNCF incubating project that defines both an open specification and a reference implementation for Kubernetes cost allocation. The project delivers two artifacts that are separate but aligned:

OpenCost Specification — a community-authored schema for what “Kubernetes cost allocation” means: how to price CPU, RAM, GPU, storage, and network at the pod level, with node-rate blending, idle cost attribution, and multi-cloud pricing provider plugs
Allocation engine — a Go implementation that conforms to the spec, scrapes kube-state-metrics and cAdvisor from Prometheus, queries cloud billing APIs for node pricing, and exposes a cost allocation REST API

The project became a CNCF incubating project in 2023 and remains there as of June 2026. It integrates with existing Prometheus installations and requires no persistent external database for its core allocation function, making it the standard zero-cost baseline for K8s cost observability.

What OpenCost covers:

Resource	Covered	Notes
Pod CPU	Yes	Time-weighted, node-rate priced
Pod RAM	Yes	Request + usage model
GPU (nvidia.com/gpu)	Yes (v1.111+)	Requires DCGM metrics or basic resource count
Persistent Volume	Yes	PVC cost via StorageClass pricing
Load Balancer	Yes	Cloud LB hourly rate
Network egress	Partial	Cross-zone/cross-region via DaemonSet traffic probe
Managed APIs (Bedrock, OpenAI)	No native	Requires Plugin (see §5)
Spot/savings plan pricing	Limited	Kubecost Enterprise does this better via CUR reconciliation

OpenCost Plugin System

The opencost/opencost-plugins repository provides the extension mechanism for non-K8s cost sources. Plugins implement a CustomCostSource Go interface — the single required method is GetCustomCosts(req CustomCostRequest) []CustomCostResponse, which returns FOCUS-spec cost rows for a given time window and resolution (hour or day). Installed plugins run as sidecar containers alongside the OpenCost pod. OpenCost polls them on the same schedule as cluster data collection.

Plugins that exist as of June 2026:

OpenAI — pulls usage and cost from the OpenAI usage API; attributes by organization/project key
Datadog — pulls infrastructure and APM costs from the Datadog cost API
MongoDB Atlas — cluster usage and cost via Atlas billing API

A Bedrock plugin does not exist in the official repo. Building one is practical: the Bedrock billing data is available via CUR 2.0 or the Cost Explorer API, and the FOCUS response schema maps cleanly to Bedrock’s service/operation/usage-type hierarchy. The implementation path is documented in §5 below.

What OpenCost Does Not Do Well

CUR reconciliation for spot and EDP pricing — OpenCost uses on-demand list prices from cloud APIs, not the actual invoice rate. Spot discounts and AWS Enterprise Discount Program adjustments are not reflected unless you run a custom billing adapter.
Multi-cluster federation — OpenCost has no native aggregation across clusters. Each cluster runs its own instance, and cross-cluster rollup requires building your own query layer over the individual APIs.
Non-engineering stakeholders — The OpenCost UI is functional but minimal. Teams expecting chargebacks, cost center reports, or budget alerting workflows typically graduate to Kubecost Enterprise.

2. Kubecost vs OpenCost Feature Comparison for AI Workloads

Feature	OpenCost (free)	Kubecost Free	Kubecost Enterprise
Pod CPU/RAM allocation	Yes	Yes	Yes
GPU cost (nvidia.com/gpu)	Yes (v1.111+)	Yes (v2.4+)	Yes
DCGM utilization-weighted GPU	No	Yes (v2.4+)	Yes
MIG partition cost allocation	No	Partial	Yes
Network cost (cross-AZ/region)	Partial	Yes	Yes
External cost plugins (OpenAI, etc.)	Yes (via plugin repo)	No	Yes (Kubecost custom cost)
CUR 2.0 reconciliation	No	No	Yes
Multi-cluster federation	No	No	Yes
Spot/savings plan accuracy	No	No	Yes
Budget alerts + Slack/PD	No	Limited	Yes
Chargeback / showback reports	No	No	Yes
FOCUS export	Yes (via plugin spec)	No	Yes
RBAC / SSO	No	No	Yes

For AI workloads specifically, the critical differentiator is DCGM-backed utilization-weighted GPU cost. Without it, cost is allocated based on nvidia.com/gpu resource requests, not actual utilization — a model-serving pod that requests 1 GPU but uses 30% of it is billed at 100% of the GPU hourly cost. Kubecost 2.4 (released 2024) changed this by ingesting DCGM_FI_DEV_GPU_UTIL from the DCGM Exporter and weighting cost allocation accordingly.

Model-serving pod cost attribution works in both tools through K8s label selectors. Pods carrying app=vllm, app=nim, or team=ml-platform can be grouped in any allocation query. Kubecost’s Allocation UI supports saving these groupings as named views with automatic refresh.

Inference cost-per-request is not a concept in either tool natively — both tools are infrastructure-layer, not request-layer. Bridging to per-request cost requires the LiteLLM sidecar pattern in §6.

3. GPU Cost Allocation Methods: Time-Slice vs MIG vs Full Node

Full GPU Node Allocation (simplest)

One pod requests and receives one GPU. nvidia.com/gpu: 1 in the pod spec. OpenCost and Kubecost both handle this correctly — the GPU node’s hourly rate is divided by the number of GPUs on the node, and the pod is billed for the fractional node cost corresponding to 1 GPU.

For a p4d.24xlarge (8× A100 80GB, ~$32.77/hr on-demand):

Per-GPU cost: $32.77 / 8 = $4.10/hr
A vLLM pod running one model-serving replica that requests 1 GPU costs $4.10/hr of GPU, plus pro-rated CPU/RAM

MIG (Multi-Instance GPU)

NVIDIA MIG partitions an A100 or H100 into up to 7 isolated GPU instances at the hardware level. Each MIG slice has dedicated memory and compute bandwidth, making it suitable for concurrent inference workloads with latency SLAs.

MIG instances are exposed in Kubernetes as fractional resources: nvidia.com/mig-1g.10gb: 1 for a 1g slice (10GB memory, 1/7 of A100 compute). Each slice is a distinct schedulable resource with its own device plugin advertising.

Kubecost MIG cost allocation (v2.x):

Kubecost Enterprise reads the MIG profile from the node label nvidia.com/mig.config and maps slice profiles to fractional GPU cost
A 1g.10gb slice on an A100 is priced at 1/7 of the per-GPU rate: $4.10/hr × (1/7) ≈ $0.59/hr per MIG slice
On a p4de.24xlarge (8× A100s, each supporting 7 MIG slices): 56 addressable MIG instances at ~$0.58/hr each

OpenCost MIG support is limited. OpenCost allocates cost based on nvidia.com/gpu requests; MIG resources use a different resource key and are not automatically costed. Teams running MIG on OpenCost need to add a custom cost adapter or use Kubecost Enterprise.

NVIDIA GPU time-slicing allows multiple containers to share a single GPU by time-multiplexing CUDA execution. Configured via the GPU Operator’s time-slicing configuration, it exposes n replicas of nvidia.com/gpu on a single physical GPU.

Cost allocation under time-slicing:

Each pod requesting nvidia.com/gpu: 1 on a time-sliced node gets billed 1/n of the GPU cost
For n=4 on a $4.10/hr GPU: each pod costs $1.025/hr for the GPU share
Kubecost handles this via the node’s configured time-slice count, which it reads from the GPU Operator device plugin configuration

Time-slice vs MIG tradeoff for cost tracking:

MIG: hard memory isolation, accurate per-slice billing, no memory contention — preferred for concurrent inference with predictable memory footprints
Time-slicing: flexible, supports arbitrary model sizes on a single GPU, no memory isolation — billing is approximate (pods can run sequentially, not concurrently in true time-slice; one idle pod inflates cost for another)
Cost attribution accuracy: MIG > time-slicing

4. NVIDIA DCGM + Kubecost: Utilization-Weighted GPU Cost Allocation

Why Resource-Request Allocation Is Insufficient

A vLLM pod serving Llama-3-70B on an A100 requests nvidia.com/gpu: 1. Without utilization weighting, it is billed the full per-GPU rate regardless of whether the GPU is at 95% utilization or 12% utilization (typical during batch inference idle periods). In practice, A100 GPU utilization in production inference clusters averages 40–65% under mixed traffic patterns, meaning request-based allocation systematically overcharges inference pods by 35–60%.

DCGM Exporter Setup

NVIDIA DCGM (Data Center GPU Manager) runs as a system-level daemon on each GPU node. dcgm-exporter is a Kubernetes DaemonSet that queries DCGM and publishes GPU metrics in Prometheus format.

Core metrics relevant to cost allocation:

DCGM_FI_DEV_GPU_UTIL — GPU compute utilization percentage (0–100), per GPU device
DCGM_FI_DEV_MEM_COPY_UTIL — memory copy utilization (proxy for memory bandwidth saturation)
DCGM_FI_DEV_FB_USED — framebuffer memory used (bytes)
DCGM_FI_DEV_POWER_USAGE — GPU power draw in watts (input to energy cost modeling)
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL — NVLink bandwidth (cross-GPU communication, relevant for distributed inference)

Kubecost + DCGM Integration Flow

Deploy dcgm-exporter DaemonSet (via NVIDIA GPU Operator Helm chart, or standalone)
Configure Kubecost to discover DCGM service via kubecostMetrics.gpuMonitoringEnabled: true and prometheus.dcgmExporter.enabled: true
Kubecost reads DCGM_FI_DEV_GPU_UTIL per device, maps device to node, calculates utilization-weighted cost
GPU cost attributed to pod = (pod’s GPU share fraction) × (GPU utilization %) × (GPU hourly rate)

Utilization floor behavior: Kubecost applies an idle cost allocation for unutilized GPU time — the same as it does for CPU idle cost. When a GPU is 30% utilized, 30% of cost is attributed to the requesting pod and 70% is classified as GPU idle cost, assignable to a shared namespace or cluster overhead bucket.

Multi-GPU pod handling: Pods requesting nvidia.com/gpu: 4 (e.g., a Megatron-LM training job) receive averaged DCGM utilization across all 4 assigned GPUs.

Prometheus Recording Rules for GPU Cost

Efficient cost computation at scale benefits from pre-computed recording rules:

# prometheus-recording-rules.yaml
groups:
  - name: gpu_cost
    interval: 1m
    rules:
      - record: kubecost:gpu_utilization_by_pod
        expr: |
          avg by (pod, namespace, node) (
            DCGM_FI_DEV_GPU_UTIL
              * on(node) group_left(pod, namespace)
            kube_pod_info
          )
      - record: kubecost:gpu_cost_per_pod_hourly
        expr: |
          kubecost:gpu_utilization_by_pod / 100
            * on(node) group_left()
          node_gpu_hourly_cost

5. Integrating Bedrock Costs into Kubecost / OpenCost

The Core Mismatch

Bedrock costs are API-call-based, billed per million tokens, with no K8s pod to attribute them to. Kubecost and OpenCost are pod-infrastructure tools. The bridge is tagging: Bedrock Application Inference Profiles carry AWS resource tags, which map to cost allocation dimensions that match K8s label conventions.

Path A: Kubecost External Cost API

Kubecost Enterprise supports an External Cost API — a sidecar endpoint that Kubecost polls on its regular collection interval. The sidecar returns JSON cost rows conforming to Kubecost’s allocation schema. Teams use this to inject Bedrock, Vertex AI, or OpenAI costs into the same Kubecost allocation views.

Implementation sketch:

# bedrock-cost-sidecar/main.py
# Simple FastAPI server that Kubecost polls at /costs
from fastapi import FastAPI
import boto3
from datetime import datetime, timedelta

app = FastAPI()
ce = boto3.client("ce", region_name="us-east-1")

@app.get("/costs")
def get_costs(start: str, end: str):
    resp = ce.get_cost_and_usage(
        TimePeriod={"Start": start, "End": end},
        Granularity="HOURLY",
        Filter={
            "Dimensions": {
                "Key": "SERVICE",
                "Values": ["Amazon Bedrock"]
            }
        },
        GroupBy=[
            {"Type": "TAG", "Key": "team"},
            {"Type": "DIMENSION", "Key": "USAGE_TYPE"}
        ],
        Metrics=["UnblendedCost"]
    )
    rows = []
    for result in resp["ResultsByTime"]:
        for group in result["Groups"]:
            team = group["Keys"][0].replace("team$", "")
            usage_type = group["Keys"][1]
            cost = float(group["Metrics"]["UnblendedCost"]["Amount"])
            rows.append({
                "name": f"bedrock/{usage_type}",
                "namespace": f"bedrock-{team}",
                "labels": {"team": team, "service": "bedrock"},
                "cost": cost,
                "start": result["TimePeriod"]["Start"],
                "end": result["TimePeriod"]["End"],
            })
    return {"costs": rows}

Deploy as a K8s Deployment + Service alongside Kubecost, then reference in Kubecost’s Helm values:

# kubecost values.yaml snippet
externalCostSources:
  - name: bedrock
    url: http://bedrock-cost-sidecar.kubecost.svc.cluster.local:8080/costs
    interval: 1h

Path B: OpenCost Bedrock Plugin

OpenCost plugins implement the CustomCostSource Go interface. A Bedrock plugin would:

Call the AWS Cost Explorer API for Amazon Bedrock line items, grouped by Application Inference Profile tags
Map each cost row to an opencost.CustomCostResponse with ResourceName, Cost, Labels, and timestamps
Return FOCUS-compliant rows that OpenCost stores alongside cluster allocation data

The plugin ships as a Docker container. OpenCost Helm chart configuration:

# opencost values.yaml - plugin config
plugins:
  enabled: true
  folder: /opt/opencost/plugin
  install:
    enabled: true
    fullImageName: "your-registry/opencost-bedrock-plugin:latest"
  configs:
    bedrock: |
      {
        "aws_region": "us-east-1",
        "cost_allocation_tag": "team",
        "granularity": "hourly"
      }

The plugin framework handles scheduling, retry, and FOCUS serialization. Teams only implement GetCustomCosts.

Athena-Based CUR Join (Alternative to Real-Time API)

For teams already exporting CUR 2.0 to S3 and querying via Athena, Bedrock costs can be joined to K8s allocation data at the team tag grain:

-- Join Bedrock CUR with OpenCost Prometheus data exported to S3
-- Step 1: Bedrock spend by team tag
WITH bedrock_spend AS (
  SELECT
    resource_tags['user:team']        AS team,
    DATE_TRUNC('hour', line_item_usage_start_date) AS hour,
    SUM(line_item_unblended_cost)     AS bedrock_cost_usd,
    SUM(line_item_usage_amount)       AS total_token_units
  FROM cur_database.cur_table
  WHERE line_item_product_code = 'AmazonBedrock'
    AND line_item_line_item_type = 'Usage'
    AND line_item_usage_start_date BETWEEN TIMESTAMP '2026-06-01' AND TIMESTAMP '2026-06-18'
  GROUP BY 1, 2
),

-- Step 2: K8s GPU + CPU spend by team label (from OpenCost allocation API export)
k8s_spend AS (
  SELECT
    labels['team']                    AS team,
    DATE_TRUNC('hour', window_start)  AS hour,
    SUM(gpu_cost + cpu_cost + ram_cost + network_cost) AS k8s_cost_usd
  FROM opencost_allocation_export  -- S3 export from OpenCost API
  WHERE window_start BETWEEN TIMESTAMP '2026-06-01' AND TIMESTAMP '2026-06-18'
  GROUP BY 1, 2
)

SELECT
  COALESCE(b.team, k.team)   AS team,
  COALESCE(b.hour, k.hour)   AS hour,
  COALESCE(b.bedrock_cost_usd, 0)  AS bedrock_usd,
  COALESCE(k.k8s_cost_usd, 0)     AS k8s_infra_usd,
  COALESCE(b.bedrock_cost_usd, 0) + COALESCE(k.k8s_cost_usd, 0) AS total_ai_cost_usd,
  COALESCE(b.total_token_units, 0) AS bedrock_tokens
FROM bedrock_spend b
FULL OUTER JOIN k8s_spend k
  ON b.team = k.team AND b.hour = k.hour
ORDER BY total_ai_cost_usd DESC;

This query produces the unified AI cost view — Bedrock API spend plus K8s inference infrastructure spend — at hourly grain, joined by the team tag/label.

6. LiteLLM Sidecar Pattern: Per-Request Cost as Prometheus Metrics

Why This Matters

Kubecost tells you that the litellm-proxy Deployment cost $4,200 in June. It cannot tell you which of your 12 internal teams drove that cost, which models were called, or what the cost-per-successful-completion was. LiteLLM’s Prometheus integration provides that request-level attribution layer.

LiteLLM Prometheus Metrics Available

LiteLLM Proxy exposes /metrics natively when PROMETHEUS_URL is set. Key metrics for cost allocation:

Metric	Type	Labels	Use
`litellm_spend_metric`	Counter	model, api_base, team_alias, user	Cumulative $ spend
`litellm_total_tokens`	Counter	model, team_alias, user	Token volume
`litellm_input_tokens`	Counter	model, team_alias	Input token breakdown
`litellm_output_tokens`	Counter	model, team_alias	Output token breakdown
`litellm_request_total_latency_metric`	Histogram	model, team_alias	p50/p95/p99 latency
`litellm_remaining_team_budget_metric`	Gauge	team_alias	Budget headroom
`litellm_provider_remaining_budget_metric`	Gauge	api_base	Provider-level budget

Kubernetes Deployment Pattern

LiteLLM proxy runs as a Kubernetes Deployment. The Prometheus integration requires no sidecar — the proxy binary exposes /metrics directly. The only configuration required is a ServiceMonitor (if using the Prometheus Operator) and setting the prometheus_url in the LiteLLM config:

# litellm-config.yaml (mounted as ConfigMap)
model_list:
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: us-east-1
  - model_name: nova-pro
    litellm_params:
      model: bedrock/amazon.nova-pro-v1:0

general_settings:
  master_key: "sk-${LITELLM_MASTER_KEY}"
  database_url: "postgresql://litellm:${DB_PASSWORD}@litellm-db:5432/litellm"

litellm_settings:
  success_callback: ["prometheus"]
  prometheus_url: "http://prometheus-operated.monitoring.svc.cluster.local:9090"
  turn_off_message_logging: true   # HIPAA/PII safety

# ServiceMonitor for Prometheus Operator scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: litellm-proxy
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: litellm-proxy
  namespaceSelector:
    matchNames:
      - litellm
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

Bridging LiteLLM Spend Metrics to Kubecost

Kubecost does not natively consume LiteLLM Prometheus metrics. The bridge is Kubecost’s custom cost metrics feature (Enterprise), which can display arbitrary Prometheus gauges/counters alongside allocation data in the UI.

For the open-source path, build a Grafana dashboard that correlates:

LiteLLM litellm_spend_metric{team_alias="payments"} — API cost by team
OpenCost container_cpu_allocation{namespace="litellm", label_team="payments"} — infrastructure cost by team
Sum = total AI cost for the payments team (infrastructure + API tokens)

# Total AI cost per team (infrastructure + API spend) — Grafana panel
(
  sum by (team_alias) (
    increase(litellm_spend_metric[1h])
  )
)
+
on(team_alias) group_left()
(
  sum by (label_team) (
    label_replace(
      container_cost_hourly{namespace="litellm"},
      "team_alias", "$1", "label_team", "(.*)"
    )
  )
)

`exporter-litellm` for PostgreSQL-Backed Spend

For teams using LiteLLM with PostgreSQL (recommended for production — the proxy stores litellm_spend_logs in Postgres), the ncecere/exporter-litellm project exports the same metrics from the database rather than from in-memory counters, providing historical backfill and more reliable aggregations across proxy restarts.

7. Networking Cost Allocation for AI Workloads

Cross-AZ Transfer in Distributed Inference

Multi-GPU inference (tensor parallelism across multiple A100 nodes) generates significant cross-AZ data transfer. AWS charges $0.01/GB for cross-AZ transfer within a region. A single forward pass in a 70B parameter model with tensor parallelism across 4 nodes can transfer 2–8 GB of activation tensors, making cross-AZ costs material at high throughput.

Kubecost network cost monitoring captures this via a DaemonSet (kubecost-network-costs) that uses Linux kernel traffic classification to intercept pod-level network flows and classify them as:

In-zone — no charge
Cross-zone (same region) — $0.01/GB (AWS)
Cross-region — $0.02–$0.09/GB depending on regions
Internet egress — $0.08–$0.09/GB

For AI workloads, enable Kubecost network cost monitoring and add the app=vllm or app=nim label selector to your allocation views to see per-model-serving-pod network cost.

Kubecost Network Cost Configuration

# kubecost values.yaml - network cost section
networkCosts:
  enabled: true
  podAnnotations: {}
  config:
    services:
      # Classify cross-AZ transfer for GPU nodes
      amazon-web-services: true
    destinations:
      - name: "cross-zone"
        subnets: ["10.0.0.0/8"]
        crossAZ: true

Practical Impact on Model Serving Architecture

When analyzing model serving costs with Kubecost network visibility, common patterns that appear:

KV-cache disaggregation (prefill on one node, decode on another) generates 1–3 GB/request of KV-cache transfer. At high QPS, cross-AZ network costs can reach 15–20% of total inference pod cost.
RAG retrieval from cross-AZ vector DB — if the vector store (Pinecone, OpenSearch) is in a different AZ than the inference pods, each retrieval adds network cost.
Recommendation: use K8s topology-aware routing (topologySpreadConstraints) to colocate inference pods and vector stores in the same AZ. Kubecost’s network cost data provides the signal to identify where this optimization is most valuable.

8. Kubecost Cluster Controller for K8s AI Platforms

What the Cluster Controller Does

The Kubecost Cluster Controller (included in Enterprise) is an in-cluster component that reads and writes K8s resources — specifically for rightsizing recommendations and automated label propagation. For AI platforms, it enables:

Automated GPU rightsizing recommendations — suggests reducing nvidia.com/gpu requests on underutilized pods, backed by DCGM utilization history
Label propagation — automatically adds cost allocation labels to pods spawned by AI platform operators (KubeFlow, Ray, Argo Workflows) that don’t carry labels by default
Continuous reconciliation — when a new NIM deployment rolls out without a team label, Cluster Controller flags it in Kubecost’s Savings dashboard as “unallocated GPU cost”

KubeFlow Cost Allocation

KubeFlow pipelines and TrainingJobs create transient pods with auto-generated names. Without label propagation, these pods appear in Kubecost as unattributed GPU cost. The recommended pattern:

Add a cost-center annotation to KubeFlow Pipeline resources
Configure Cluster Controller’s label injection to propagate cost-center to all pods created by those pipelines
Use Kubecost’s Allocation view grouped by cost-center to see training job costs

KubeFlow Pipeline pod naming convention: {pipeline-name}-{run-id}-{step-name}-{random}. Kubecost groups by the pipeline-name label if label propagation is configured correctly.

Ray Cluster Cost Attribution

Ray on Kubernetes (KubeRay operator) creates Ray head pods and worker pods. Ray Tasks are not visible at the K8s level — only the worker pod that executed them is. Cost attribution flow:

Ray worker pods: label with ray.io/cluster and ray.io/node-type (auto-applied by KubeRay)
Kubecost groups by ray.io/cluster to get per-Ray-cluster cost
Per-task cost requires Ray’s own metrics (e.g., ray_tasks_running and a custom cost rate calculation outside Kubecost)

NVIDIA NIM Cost Tracking

NVIDIA NIM containers ship with standard K8s labels from the NIM Operator: nvidia.com/nim.model, nvidia.com/nim.version. Kubecost picks these up automatically in label-based allocation views, giving per-model cost breakdown across all NIM deployments without any manual labeling.

# Kubecost Allocation API query for NIM model costs
# GET /model/allocation?window=7d&aggregate=label:nvidia.com/nim.model
# Returns: cost breakdown by deployed NIM model over the last 7 days

9. Self-Hosted GPU Inference vs Bedrock: Kubecost Break-Even Analysis

Cost Components

Self-hosted GPU inference on EKS:

GPU instance cost (EC2): largest component
EKS cluster overhead (control plane + node group management)
Kubecost + observability stack: ~$0 (OSS) to $2,000–5,000/month (Enterprise, per cluster)
Engineering operational cost: $3,000–8,000/month effective DevOps burden
Networking: varies with cross-AZ pattern

Bedrock on-demand:

Token cost: varies by model. Claude 3.5 Sonnet: $3.00 input / $15.00 output per million tokens. Amazon Nova Pro: $0.80 input / $3.20 output per million tokens. (Pricing as of June 2026; verify at aws.amazon.com/bedrock/pricing.)
No infrastructure cost, no operational burden
Higher per-token cost at volume vs. self-hosted open-weight models

Break-Even Calculation

For a team considering self-hosting Llama-3-70B on 2× A100 nodes vs. using Bedrock Nova Pro:

Parameter	Self-Hosted (2× p4d.24xlarge)	Bedrock Nova Pro
Monthly EC2 cost (on-demand)	$47,000 (2 × $32.77/hr × 720)	$0
Monthly EC2 cost (1-year RI)	~$28,000	$0
Engineering overhead	$4,000/month	$0
Effective monthly base	~$32,000	$0
Token cost at 50M tokens/month	$0 (amortized in EC2)	$200–400 (Nova Pro)
Token cost at 500M tokens/month	$0	$2,000–4,000
Token cost at 5B tokens/month	$0	$20,000–40,000
Break-even volume	~4B tokens/month	—

Self-hosting wins above ~4B tokens/month when using 1-year Reserved Instances and treating engineering cost as fixed. Below that volume, Bedrock’s operational simplicity and pay-per-token model is lower total cost.

Kubecost break-even analysis workflow:

Tag all inference pods with model=llama-3-70b, query Kubecost Allocation API for 30-day GPU + CPU + network cost
Pull Bedrock CUR data for the same period and any equivalent Nova Pro usage (or estimate from token projections)
Compare: if Kubecost shows self-hosted infra cost > projected Bedrock cost at your token volume, migrate or reduce replica count
Use Kubecost Savings recommendations to identify GPU idle time — if average utilization is below 40%, self-hosting economics worsen

The defilantech/infercost open-source tool provides a complementary analysis: it computes true cost-per-token from GPU amortization, electricity, and real power draw via DCGM DCGM_FI_DEV_POWER_USAGE, useful for on-premises GPU fleets where EC2 pricing doesn’t apply.

10. OpenCost FOCUS Export: Joining with Bedrock FOCUS Data

FOCUS 1.4 Status (June 2026)

The FinOps Open Cost and Usage Specification reached version 1.4 in June 2026. FOCUS 1.4 adds InvoiceDetail and BillingPeriod datasets for AP/finance reconciliation. AWS supports FOCUS 1.2 via CUR 2.0 exports to S3 as Parquet files.

Key FOCUS columns relevant to AI cost joins:

Column	OpenCost usage	Bedrock CUR usage
`ServiceName`	“Kubernetes” (custom)	“Amazon Bedrock”
`ResourceId`	pod/namespace identifier	Model invocation resource ARN
`ResourceType`	“Pod”, “PersistentVolume”	“AWS::Bedrock::InferenceProfile”
`Tags`	K8s labels as tag map	AWS resource tags
`BilledCost`	Allocated pod cost	Bedrock unblended cost
`UsageType`	“CPU-Hours”, “GPU-Hours”	“Tokens-Input”, “Tokens-Output”
`ChargePeriodStart`	Hour start	Hour start

OpenCost FOCUS Export via Plugin Framework

OpenCost plugins return CustomCostResponse objects that conform to FOCUS. When running an OpenAI plugin alongside K8s cost data, both datasets are tagged with ServiceName and Tags, enabling FOCUS-compliant joins in any analytics tool.

For Bedrock, the join key is Tags["team"] appearing in both:

OpenCost allocation data: K8s pod label team
Bedrock FOCUS export: AWS resource tag team on Application Inference Profiles

Athena-Native FOCUS Join

If both datasets are in S3 (CUR 2.0 Parquet + OpenCost FOCUS export), use Athena’s FOCUS 1.2 schema directly:

-- Unified AI cost view: K8s infra + Bedrock API, FOCUS schema
SELECT
  COALESCE(k.tags['team'], b.tags['user:team'])  AS team,
  DATE_TRUNC('day', k.charge_period_start)        AS day,
  SUM(CASE WHEN k.service_name = 'Kubernetes'
           THEN k.billed_cost ELSE 0 END)         AS k8s_infra_cost,
  SUM(CASE WHEN b.service_name = 'Amazon Bedrock'
           THEN b.billed_cost ELSE 0 END)         AS bedrock_api_cost,
  SUM(k.billed_cost) + SUM(b.billed_cost)         AS total_ai_cost
FROM opencost_focus_export k
FULL OUTER JOIN aws_focus_export b
  ON k.tags['team'] = b.tags['user:team']
  AND DATE_TRUNC('hour', k.charge_period_start)
    = DATE_TRUNC('hour', b.charge_period_start)
WHERE k.charge_period_start >= DATE '2026-06-01'
  OR b.charge_period_start >= DATE '2026-06-01'
GROUP BY 1, 2
ORDER BY total_ai_cost DESC;

Note: AWS CUR 2.0 uses user:team tag prefix; OpenCost typically exports bare team. Normalize at ingestion or in the query as shown.

11. Helm Chart Configuration: Kubecost with GPU Support and Bedrock External Cost

Full Kubecost values.yaml for AI Workloads

# kubecost-values-ai.yaml
# Kubecost Enterprise — AI/ML workload configuration
# Apply with: helm upgrade --install kubecost kubecost/cost-analyzer \
#   -n kubecost --create-namespace \
#   -f kubecost-values-ai.yaml

global:
  prometheus:
    enabled: true
    fqdn: http://prometheus-operated.monitoring.svc.cluster.local:9090
  grafana:
    enabled: false      # use existing Grafana
    proxy: false

kubecostMetrics:
  # Enable GPU cost tracking via DCGM
  gpuMonitoringEnabled: true

prometheus:
  # DCGM Exporter integration
  dcgmExporter:
    enabled: true
    # If DCGM Exporter already installed, provide its service endpoint
    serviceAddress: "http://dcgm-exporter.gpu-system.svc.cluster.local:9400"
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: prometheus

# Network cost DaemonSet for cross-AZ tracking
networkCosts:
  enabled: true
  podAnnotations: {}
  config:
    services:
      amazon-web-services: true
    logLevel: "warn"

# CUR 2.0 reconciliation (Enterprise)
awsStore:
  enabled: true
  createServiceAccount: false   # use IRSA
  serviceAccountAnnotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/kubecost-cur-reader
  bucket: "s3://my-cur-bucket"
  region: "us-east-1"
  athenaDatabase: "athenacurcfn_cur_database"
  athenaTable: "cur_table"
  athenaWorkgroup: "kubecost"
  projectID: "123456789012"

# External cost sidecar for Bedrock
# Deploy bedrock-cost-sidecar separately (see §5), then reference here
# (Kubecost Enterprise feature — configure via UI or API post-install)

# Savings + rightsizing
clusterController:
  enabled: true          # enables label propagation + GPU rightsizing
  fqbn: ""

# Allocation defaults
allocationDefaults:
  # Unallocated GPU time goes to shared-overhead namespace
  idleCostAllocationTarget: "shared-overhead"

# RBAC — namespace-level access for team billing visibility
kubecostNetworkCosts:
  enabled: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/kubecost-main

podAnnotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

DCGM Exporter DaemonSet (if not using GPU Operator)

# dcgm-exporter-values.yaml
# helm install dcgm-exporter nvidia/dcgm-exporter \
#   -n gpu-system --create-namespace \
#   -f dcgm-exporter-values.yaml

image:
  repository: nvcr.io/nvidia/k8s/dcgm-exporter
  tag: "3.3.8-3.6.0-ubuntu22.04"

serviceMonitor:
  enabled: true
  additionalLabels:
    release: prometheus
  interval: 15s

nodeSelector:
  nvidia.com/gpu.present: "true"

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

# Custom metrics: add power draw for energy cost modeling
customMetrics: |
  # FIELDS
  DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %)
  DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %)
  DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB)
  DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB)
  DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W)
  DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, gauge, NVLink bandwidth (MB/s)
  DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Fraction of time GR engine active

OpenCost with Plugin Support (for OpenAI / custom Bedrock plugin)

# opencost-values-ai.yaml
# helm install opencost opencost/opencost \
#   -n opencost --create-namespace \
#   -f opencost-values-ai.yaml

opencost:
  exporter:
    defaultClusterId: "eks-prod-us-east-1"
    cloudProviderApiKey: ""   # not needed for AWS (uses IRSA)

  ui:
    enabled: true

  prometheus:
    external:
      enabled: true
      url: "http://prometheus-operated.monitoring.svc.cluster.local:9090"

  # Plugin framework
  plugins:
    enabled: true
    install:
      enabled: true
    configs:
      # OpenAI plugin (ships in opencost/opencost-plugins)
      openai: |
        {
          "openai_api_key": "${OPENAI_API_KEY}",
          "aggregate_by": "project"
        }
      # Custom Bedrock plugin (build using opencost-plugins framework)
      # bedrock: |
      #   {
      #     "aws_region": "us-east-1",
      #     "cost_allocation_tag": "team",
      #     "granularity": "hourly"
      #   }

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/opencost-reader

Summary: Decision Matrix

Need	Recommended Tool	Notes
Free K8s cost allocation	OpenCost	CNCF incubating, Prometheus-native
GPU cost by request (no DCGM)	OpenCost v1.111+ or Kubecost free	Allocates by resource request, not utilization
GPU cost utilization-weighted	Kubecost v2.4+ (any tier)	Requires DCGM Exporter deployed
MIG partition billing	Kubecost Enterprise	Reads MIG profile from GPU Operator labels
Bedrock cost in same dashboard	Kubecost Enterprise (External Cost API) or OpenCost plugin (DIY)	No off-the-shelf Bedrock plugin for OpenCost today
Per-request LLM cost attribution	LiteLLM Prometheus metrics + Grafana	Kubecost does not do request-level attribution
Multi-cluster AI spend rollup	Kubecost Enterprise	OpenCost requires custom aggregation layer
Break-even: self-hosted vs Bedrock	Kubecost + CUR 2.0 Athena query	See §9 Athena SQL
FOCUS-compliant unified cost view	OpenCost plugins + AWS CUR 2.0	Join on team tag; see §10 SQL
KubeFlow/Ray job cost labeling	Kubecost Cluster Controller	Auto-propagates labels to transient pods