← Agent Frameworks 🕐 17 min read
Agent Frameworks

Mid-Market AI FinOps — Lightweight Spend Governance Without a Dedicated Team (2026)

1. Why Enterprise FinOps Tooling Is Overkill Below $50K/Month · March 2026

Enterprise FinOps tools were built for organizations spending $1M+/month on cloud. The overhead is real:

Audience: CTO, technical co-founder, or VP Engineering at a $50M–$500M revenue company running AI workloads on AWS (Bedrock), OpenAI, Anthropic, or some mix. You have no FinOps team. You may not have a cloud cost analyst. You need AI spend under control before your CFO asks a question you can’t answer.

Bottom line up front: At under $50K/month AI spend, the enterprise FinOps toolkit — CUDOS dashboards, FOCUS data exports, Athena cost views, dedicated tooling like Apptio or CloudZero — costs more to run than it saves. The right model is a three-tag standard, one Budget Alert, a weekly Cost Explorer check, and a Google Sheets tracker that your finance person can read. That gets you 80% of the governance value at 5% of the overhead.

The enterprise FinOps stack and what it actually costs

Enterprise FinOps tools were built for organizations spending $1M+/month on cloud. The overhead is real:

  • CUDOS (AWS Cost and Usage Dashboard): AWS’s own best-practice CUR dashboard requires enabling Cost and Usage Reports, creating an S3 bucket, running Glue crawlers, and setting up Athena queries. Baseline setup is 2–3 days of engineer time. Ongoing: someone has to understand Athena SQL and maintain the Glue schema as AWS adds new line items. At $10K/month AI spend, the engineering time to maintain this exceeds its value.
  • FOCUS exports: The FinOps Open Cost and Usage Specification is the emerging standard for multi-cloud cost normalization. Excellent idea for a company running Azure + AWS + GCP at scale. For a mid-market company with 80% of AI on AWS Bedrock and the rest on OpenAI API, the normalization problem doesn’t exist yet.
  • Commercial FinOps platforms (CloudZero, Apptio Cloudability, Spot.io): Licensing starts at $2K–$5K/month for small accounts. At $20K/month AI spend, you’re paying 10–25% of your AI cost on a tool to track your AI cost.
  • Dedicated FinOps headcount: A FinOps engineer costs $130K–$180K/year fully loaded. The math only works above roughly $500K/month cloud spend — at that scale, 3–5% savings from optimization pays for the role.

What the mid-market actually needs

At $50K/month or below, the governing questions are simple:

  1. Which team or product is spending the most?
  2. Is any single workload running away?
  3. Are we on track vs. budget this month?
  4. Which model or service is the biggest line item?

These four questions can be answered with AWS Cost Explorer (free, included in every AWS account), three resource tags, and one Budget Alert. No Athena. No Glue. No SQL.

2. Minimum Viable AI Cost Governance

The 3-tag standard

Tagging is the foundation. Without tags, Cost Explorer gives you aggregate spend by service (e.g., “Amazon Bedrock: $8,432 this month”) but cannot tell you which team, product, or application is responsible. With three tags consistently applied, you get team-level attribution without any additional tooling.

Required tags — enforce these and only these to start:

Tag key Example values Purpose
team data-platform, product-eng, cx-automation Owner attribution — who gets the cost
environment prod, staging, dev Separate production costs from experiments
application customer-support-bot, contract-analyzer, internal-qa Per-use-case attribution

Enforcement: Apply tags at the IAM role or resource level. For Bedrock, tag the IAM role each service uses. For OpenAI/Anthropic API calls, you can’t tag at the provider level — pass cost attribution through your own logging (covered in section 6).

Practical rule: If a resource doesn’t have all three tags, it goes into an untagged bucket that you review weekly. The goal isn’t 100% tag coverage immediately — it’s making untagged spend visible and embarrassing enough that engineers fix it.

One Budget Alert

AWS Budgets is free for the first two budgets per account. Set one now:

  1. Go to AWS Budgets → Create Budget
  2. Choose Cost budget
  3. Set the monthly amount to 120% of your expected AI spend (gives a buffer before the alert fires)
  4. Filter by service: Amazon Bedrock (add others as needed)
  5. Alert at 80% of budget (forecasted) and 100% (actual)
  6. Send to: CTO + finance email alias

That’s it. One alert. If it fires at 80% with two weeks left in the month, you have time to investigate and act. If you wait until 100% actual, the damage is done.

Weekly Cost Explorer check — 15 minutes

Every Monday morning, open Cost Explorer (AWS Console → Cost Management → Cost Explorer) and run three views:

View 1: Monthly spend by service, current month vs. prior month

  • Filter: last 3 months
  • Group by: Service
  • Look for: any service with >20% month-over-month increase

View 2: Bedrock spend by model

  • Filter: Service = Amazon Bedrock, current month
  • Group by: Usage type (this approximates model-level breakdown)
  • Look for: which model family is the biggest cost driver

View 3: Spend by team tag

  • Filter: current month
  • Group by: Tag = team
  • Look for: untagged spend percentage, any team exceeding their informal allocation

Total time: 15 minutes. Record the numbers in your tracker (section 3). If anything looks anomalous, dig in — otherwise close the tab.

3. Spreadsheet-First: Monthly AI Cost Tracker

Build this once in Google Sheets or Excel. Update it on the first Monday of each month from Cost Explorer CSV exports.

How to export from Cost Explorer

  1. Run the view you want (e.g., monthly spend by service)
  2. Click Download CSV (top right)
  3. Open in Sheets, paste into the raw data tab

Tracker structure

Tab 1: Raw — Monthly Totals

Month Bedrock OpenAI Anthropic API Other AI Total AI Spend Budget
2026-01 $4,200 $1,800 $600 $0 $6,600 $8,000
2026-02 $5,100 $2,200 $800 $0 $8,100 $8,000

Tab 2: By Team

Pull from Cost Explorer group-by-tag export. One row per team per month. Add a % of total column so you can see concentration at a glance.

Tab 3: Unit Economics

This is the column that matters most to your CFO:

Month Total AI Spend Active Users / Requests / Units Cost per Unit MoM Change
2026-01 $6,600 45,000 API calls $0.147 —
2026-02 $8,100 62,000 API calls $0.131 -11%

If cost per unit is falling as volume grows, your model is working. If cost per unit is rising, you have an efficiency problem — likely a model choice issue or a prompt getting longer.

Tab 4: Anomalies and Actions

Free text. Date, what was noticed, what was done, who owns follow-up. This becomes your audit trail if a CFO asks why March was 40% over budget.

Who updates it

Finance owns Tab 1 (they pull the monthly total). Engineering owns Tabs 2 and 3 (they know what the unit denominators mean). Review Tab 4 together in a monthly 30-minute sync.

4. When to Graduate: Spend Thresholds for Tooling Investment

The lightweight model has a ceiling. Here are the breakpoints where investing in more infrastructure pays off.

$10K/month — First governance checkpoint

At this level, add:

  • A second Budget Alert scoped to a specific team or application (e.g., your biggest spender)
  • Simple cost logging in your application: log model, token count, and team to CloudWatch. Write a weekly Lambda that queries CloudWatch Insights and posts a summary to Slack. This gives you application-level visibility that Cost Explorer cannot.

Still no need for Athena, CUDOS, or commercial tooling.

$50K/month — Intermediate governance

At this level, add:

  • Enable Cost and Usage Reports (CUR) to S3. You don’t need to query it yet, but having the raw data lets you backfill analysis later.
  • Evaluate LiteLLM proxy (covered in section 6). At $50K/month, the visibility and routing control it provides starts to justify its operational overhead.
  • Add a designated cost owner: one engineer who spends 2–4 hours per week on AI cost analysis, not just the CTO doing a Monday check.

Still not ready for commercial FinOps platforms unless you’re multi-cloud.

$200K/month — Serious infrastructure warranted

At this level:

  • CUDOS or a commercial FinOps platform makes economic sense. Even 3% optimization savings is $6K/month, which covers tooling costs.
  • A dedicated FinOps engineer role is justified if spend is growing.
  • EDP negotiation becomes realistic (covered in section 9).
  • FOCUS exports matter if you’re genuinely multi-cloud.

The key insight: don’t jump to the $200K/month governance model at $20K/month spend. The overhead will slow your team down without proportional benefit.

5. The 3-Person FinOps Lite Model

For a mid-market company, AI cost governance does not require a dedicated function. It requires clear role ownership across three existing roles.

Role 1: CTO (or VP Engineering) — Accountable owner

Time commitment: 30 minutes per week, 1 hour per month for review

Responsibilities:

  • Sets the AI spending budget (in collaboration with CFO)
  • Reviews the weekly Cost Explorer summary
  • Escalates anomalies to team leads
  • Makes model selection decisions when cost vs. quality tradeoffs arise (e.g., “do we need GPT-4o for this use case or does Claude Haiku work?”)
  • Approves any new AI service or model before it goes to production

Not responsible for: Day-to-day cost monitoring, tagging compliance enforcement, vendor negotiations (until EDP territory)

Role 2: Designated engineer — Operational owner

Pick one engineer who cares about systems efficiency. This is not a full-time job at mid-market scale.

Time commitment: 2–4 hours per week

Responsibilities:

  • Maintains the tagging standard and chases down untagged resources
  • Runs the weekly Cost Explorer check and updates the tracker
  • Owns the cost logging infrastructure (CloudWatch, or equivalent)
  • Reviews token usage patterns and flags inefficiencies (e.g., prompts that have grown to 8,000 tokens when 2,000 suffices)
  • Owns the Budget Alert response: when an alert fires, this person investigates within 24 hours

Key skill: Comfortable reading AWS Cost Explorer. Does not need to know Athena, Terraform, or FinOps certifications.

Role 3: Finance (controller or FP&A analyst) — Budget owner

Time commitment: 2 hours per month

Responsibilities:

  • Maintains Tab 1 of the cost tracker (monthly totals)
  • Maps AI spend to departmental cost centers for P&L purposes
  • Flags when AI spend is trending against budget before month-end
  • Produces the quarterly AI cost summary for the CFO

Meeting cadence:

  • Weekly: CTO reviews Monday Cost Explorer summary alone (15 min)
  • Monthly: CTO + designated engineer + finance for 30-min tracker review
  • Quarterly: CTO + CFO for AI spend vs. plan, with unit economics trend

That’s the entire governance structure. Three people, defined roles, one monthly meeting, one quarterly meeting.

6. LiteLLM for Mid-Market: Is It Worth It Below $10K/Month?

LiteLLM is an open-source proxy that sits between your application and LLM providers. It adds a unified API layer, usage tracking, budget limits per user or team, and model routing (e.g., fall back to a cheaper model if the primary is unavailable).

What it gives you

  • Per-user and per-team spend tracking without AWS tags (useful for OpenAI/Anthropic where you can’t tag at the infrastructure level)
  • Hard spend limits: if a team exceeds their monthly budget, requests start failing — useful guardrail
  • Model routing: route cost-sensitive workloads to cheaper models automatically
  • A dashboard: built-in spend visibility that doesn’t require Cost Explorer

The operational cost

LiteLLM requires a running server (typically an EC2 instance or ECS task), a PostgreSQL database, and ongoing maintenance as the library updates. The infrastructure adds latency (typically 10–50ms) to every API call. Someone has to own upgrades, SSL certs, and availability.

At sub-$10K/month: The spreadsheet tracker plus Cost Explorer gives you 70% of what LiteLLM provides, with zero infrastructure to maintain. Unless you have a specific need — per-user billing, hard budget cutoffs, or provider redundancy — skip LiteLLM until you hit $10K–$20K/month.

At $10K–$50K/month: LiteLLM starts making sense if you have multiple teams or products generating AI spend that you can’t attribute through AWS tags alone (i.e., you’re spending on OpenAI or Anthropic direct APIs, not just Bedrock).

At $50K+/month: LiteLLM or a comparable proxy (Portkey, MLflow AI Gateway) is worth the operational overhead. The routing and limits features provide real cost control that passive monitoring cannot.

7. AWS Budget Actions as a Hard Stop

Budget Alerts tell you when you’ve hit a threshold. Budget Actions do something about it. For mid-market companies, this is the simplest hard governance control available.

How it works

AWS Budgets can automatically attach an IAM policy when a threshold is crossed. The policy denies Bedrock API calls, effectively shutting off AI spend if you exceed your limit.

Setup

  1. Go to AWS Budgets → your existing budget → Manage Alerts
  2. Add a Budget Action on the 100% actual threshold
  3. Action type: Apply IAM Policy
  4. Create a deny policy for Bedrock:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*"
    }
  ]
}
  1. Attach to: the IAM roles used by your AI workloads
  2. Approval: you can require manual approval (CTO gets an email and must confirm) or auto-apply

Recommendation for mid-market: Use manual approval on the action. Fully automatic shutoff can break production workflows in ways that create more cost (support escalations, customer refunds) than the spend you were avoiding. The value of the action is that it forces a deliberate decision — someone has to click “approve” and acknowledge the situation.

Set the budget action threshold at 110–120% of your monthly budget, not 100%. This prevents false positives from legitimate month-end spikes.

What it doesn’t cover

Budget Actions work for AWS services. They do not affect OpenAI or Anthropic API keys. For those, use per-key spend limits in the provider’s dashboard (OpenAI and Anthropic both support this). Set a hard monthly limit on every API key.

8. Cost Allocation Without Athena

Cost Explorer’s Group By feature gives you team-level attribution without writing a single SQL query. Here’s how to use it for the attribution reporting your CFO wants.

Group by tag in Cost Explorer

  1. Cost Explorer → Explore costs
  2. Set time range to current month (or last month for reporting)
  3. Group by: Tag → select team
  4. Filter: Service = Amazon Bedrock (add other AI services as separate views)

This gives you a bar chart and table showing Bedrock spend broken down by team tag. Download as CSV. Paste into your tracker.

Handling untagged spend

Untagged resources appear as No tag key in the group-by view. Your goal is to drive this below 5% of total spend. To investigate untagged Bedrock spend:

  1. Switch Group By from Tag to Resource (if Bedrock supports resource-level tagging in your region)
  2. Or switch to Usage Type to understand what kind of usage is untagged (inference, embedding, etc.)
  3. Cross-reference with CloudWatch logs to identify which application or Lambda function is generating the calls

Practical tip: Add tag enforcement to your CI/CD pipeline. If a Terraform or CDK deployment creates a Bedrock resource without the three required tags, the pipeline fails. This prevents untagged resources from appearing in the first place rather than chasing them after the fact.

Multi-provider attribution without SQL

For OpenAI and Anthropic spend (which you can’t tag at the AWS level), log the cost proxy in your application:

  • On every API call, log: timestamp, team, application, model, input tokens, output tokens, estimated cost
  • Write to CloudWatch Logs with a structured JSON format
  • Use CloudWatch Logs Insights (no setup required, pay per query) for monthly attribution queries

Example CloudWatch Insights query:

fields @timestamp, team, application, model, estimatedCost
| stats sum(estimatedCost) as totalCost by team, application
| sort totalCost desc

This gives you the same team-level attribution for OpenAI spend that you get from Cost Explorer tags for Bedrock spend. No Athena, no Glue, no S3 bucket setup.

9. Mid-Market Vendor Negotiation

When to ask for an EDP

An AWS Enterprise Discount Program (EDP) commitment requires a minimum spend commitment of $100K–$500K over 1–3 years. At mid-market AI spend levels:

  • Under $200K/year AI on AWS: Don’t bother with EDP. The discount (typically 5–15%) doesn’t justify the commitment lock-in. Use On-Demand and retain flexibility.
  • $200K–$500K/year: Ask for an EDP conversation. AWS will typically offer 8–12% discount at this level. Worth it if your AI roadmap is clear for 18+ months.
  • $500K+/year: EDP is warranted. Engage AWS Enterprise Sales. Typical discounts at this level: 12–20% depending on commitment term and growth rate. Bring your 12-month cost history and a 24-month forecast.

For Bedrock specifically: AWS offers Provisioned Throughput for Bedrock models. At high, consistent volume, Provisioned Throughput (where you commit to model units for 1 or 6 months) is cheaper than On-Demand. The break-even is roughly 70% utilization of the committed capacity. Don’t provision unless you can forecast demand accurately enough to hit that utilization.

OpenAI and Anthropic direct negotiation

Both OpenAI and Anthropic offer enterprise agreements with committed spend discounts:

  • OpenAI: Volume discounts start at $100K/year commitments. Typical discounts: 10–20% depending on model mix and volume.
  • Anthropic: Similar structure. Claude API discounts available at $100K+/year. Worth asking for a rate review conversation at $50K/year.

Negotiating posture for mid-market: You have more leverage than you think if you’re growing. Lead with your growth rate, not your current spend. “We spent $40K last year and we’re on track for $180K this year” is a stronger position than “we want a discount on $40K spend.”

10. Common Mid-Market AI Cost Mistakes

These are the issues that cause surprise cost spikes at mid-market companies. Most are fixable in a day.

1. Not setting max_tokens on API calls

Every LLM API call should have a max_tokens parameter. Without it, the model can generate up to its context window limit. A prompt that asks for a summary and gets a 4,000-token response when 500 suffice is wasting 87% of its output budget.

Fix: Set max_tokens on every call. For summarization tasks, 500–800 tokens. For generation tasks, calibrate to actual need. Review your top 5 most frequently called endpoints and add this parameter today.

2. Using GPT-4o (or Claude Opus) for everything

The highest-capability model is not the right choice for every task. Input classification, intent detection, simple Q&A, structured data extraction — these tasks perform nearly identically on smaller, cheaper models.

Rough cost differential (2026):

  • GPT-4o vs. GPT-4o-mini: 15–20x cost difference per token
  • Claude Opus vs. Claude Haiku: 10–15x cost difference per token

Fix: Audit your top 3 workloads by spend. For each, run a 100-call sample through a smaller model and score the outputs. If accuracy is within 5%, switch. Most mid-market companies find that 60–70% of their AI calls can run on smaller models without user-visible quality loss.

3. No tagging on new resources

Every new AI workload added without tags immediately creates unattributed spend. This isn’t a compliance problem — it’s a visibility problem. You cannot manage what you cannot see.

Fix: Add a pre-deployment checklist. Before any new AI workload goes to production: three tags applied, IAM role scoped to minimum necessary permissions, max_tokens set, Budget Alert verified to cover the new spend.

4. Surprise costs from RAG re-ingestion

Retrieval-Augmented Generation systems require embedding your document corpus. Embedding is charged per token. The problem: many teams re-embed their entire corpus on every deployment or nightly cron job, when they should only embed changed documents.

Fix: Implement delta ingestion. Track document hashes. Only re-embed documents that have changed since last ingestion. For a 10,000-document corpus, this can reduce embedding costs by 90–95% month-over-month once the initial corpus is embedded.

Additionally, watch for runaway re-ingestion jobs. A bug that causes a loop can embed a corpus hundreds of times in a single run. Set a Budget Alert scoped specifically to embedding usage type.

5. Development and staging environments hitting production endpoints

Developers testing against Claude or GPT-4o in their local environments, or staging environments running full traffic simulations, can generate significant unplanned spend.

Fix:

  • Use environment tag to separate prod vs. dev/staging spend
  • Set per-environment Budget Alerts (not just per-account)
  • For development: use cheaper models or local models (Ollama, llama.cpp) for all testing that doesn’t require production model quality
  • For staging: gate staging environment access behind a spend limit in LiteLLM or at the API key level

6. Not reviewing the cost per output unit monthly

AI costs in isolation are hard to manage. AI cost per unit of output (per user, per document processed, per query answered) tells you whether your efficiency is improving or degrading over time.

A workload that costs $2,000/month and processes 20,000 documents is efficient. The same workload costing $4,000/month and processing 22,000 documents has a problem. Without the unit metric, you see a cost increase and assume it’s growth. With the unit metric, you see the 60% cost-per-document increase and know something broke.

Fix: Define the unit for every major AI workload. Track cost-per-unit monthly in your tracker. Make it the primary metric, not raw spend.

Summary: The Minimum Viable AI FinOps Stack for Mid-Market

Do this now (if not done):

  • [ ] Apply three tags to all AI resources: team, environment, application
  • [ ] Set one AWS Budget Alert at 80% and 100% of your monthly AI budget
  • [ ] Set API key spend limits in OpenAI and Anthropic dashboards
  • [ ] Add max_tokens to all API calls

Do this this month:

  • [ ] Build the Google Sheets cost tracker (Tabs 1–4 above)
  • [ ] Run the weekly Cost Explorer check for 4 consecutive weeks — you’ll spot your first anomaly within a month
  • [ ] Identify your top 3 workloads by spend and audit model selection

Do this this quarter:

  • [ ] Add cost logging to your application for non-Bedrock providers
  • [ ] Audit your RAG ingestion jobs for unnecessary re-embedding
  • [ ] Set up the monthly 30-minute governance meeting

Revisit at these thresholds:

  • $10K/month: Add application-level cost logging
  • $50K/month: Evaluate LiteLLM proxy, enable CUR to S3
  • $200K/month: Evaluate commercial FinOps tooling, pursue EDP

The goal is not FinOps as a discipline. The goal is being able to answer your CFO’s question — “how much are we spending on AI, and is it working?” — without a three-week investigation. The above gets you there in a week of setup and 30 minutes per week to maintain.

Sources and methodology: AWS Budgets and Cost Explorer documentation (AWS, 2026); LiteLLM project documentation (BerriAI, 2026); FinOps Foundation maturity model (FinOps Foundation, 2025); Bedrock Provisioned Throughput pricing (AWS, 2026); OpenAI and Anthropic enterprise pricing tiers (provider documentation, 2026). Unit cost estimates are indicative and vary by region, negotiated rates, and model version.