← Benchmarks 🕐 9 min read
Benchmarks

Enterprise Eval for Fine-Tuned SLMs and RAG Pipelines

Brandon Sneider · May 2026

> **Source credibility:** GitHub repo data is TIER 1 (primary source, observed directly). Peer-reviewed papers are TIER 1. Vendor documentation is TIER 2.

See also (wiki): wiki/enterprise-rag-architecture.md · wiki/agentic-ai-governance.md · wiki/ai-model-evaluation-benchmarks.md · wiki/enterprise-slm-specialization.md · wiki/slm-fine-tuning-enterprise.md

Cross-references within pillar: slm-data-curation-pipeline-2026.md · enterprise-slm-finetuning-oss-landscape-2026.md

Source credibility: GitHub repo data is TIER 1 (primary source, observed directly). Peer-reviewed papers are TIER 1. Vendor documentation is TIER 2. Vendor marketing claims about customer counts or eval volume are TIER 3 unless corroborated.

Why This Note Exists

Pillar 21 covers benchmarks for frontier models extensively. What was missing: how enterprises evaluate SLMs they’ve fine-tuned themselves — not against public benchmarks, but against the specific task they trained for. This note covers: RAGAS for RAG pipelines, CI/CD eval gating patterns, LLM-as-Judge for domain rubrics, off-the-shelf domain eval datasets, and the pre-deployment eval checklist.

The safety finding that changes the checklist: Multiple 2025 papers confirm that fine-tuning on benign datasets partially removes safety guardrails from aligned base models. The mechanism is documented: alignment occupies sparse parameter regions that downstream optimization displaces. Fine-tuning on GSM8K or Alpaca datasets has been shown to increase harmful output rates. This is not a theoretical risk — it is a confirmed, reproducible failure mode that should be a first-class eval category.

RAGAS — The De Facto RAG Evaluation Standard

GitHub: github.com/explodinggradients/ragas | Stars: ~13,300 | License: Apache-2.0 | Backer: YC W24

Reference-free evaluation of RAG pipelines. Does not require labeled ground truth to run core metrics.

Core Metrics

Metric What it measures LLM call?
Faithfulness Does the answer contain only claims supported by retrieved context? Yes — decompose + verify
Answer Relevancy Does the answer address the question? Yes — generate hypothetical questions, embed-compare
Context Precision Are retrieved chunks ranked with most relevant first? Yes
Context Recall Does retrieved context cover everything needed to answer? Yes — requires reference answer

Mechanism. All four core metrics use LLM-as-Judge internally. Faithfulness decomposes the answer into atomic claims, then verifies each against context via a second LLM prompt. Every RAGAS run approximately doubles LLM API cost relative to the original query.

Cost to run. From official RAGAS cost documentation: evaluating 20 samples with GPT-4o ≈ 25,097 input + 3,757 output tokens ≈ $1.17. Generating a 10-item test set from a knowledge graph adds ~$0.21. At 1,000 eval samples: budget $50–80 with GPT-4o; swap judge to gpt-4o-mini to drop cost ~15× with some accuracy loss on subtle hallucinations. [TIER 1 — official docs]

Industry thresholds in documented practice: faithfulness ≥ 0.85 for customer-facing deployments, ≥ 0.90 for regulated domains (finance, healthcare, legal). [TIER 2 — industry blog consensus, not peer-reviewed]

Enterprise adoption. AWS, Microsoft, Databricks, Moody’s cited in marketing materials; 5M+ evaluations/month reported. [TIER 3 — vendor-reported, unaudited]

Critical Limitation

RAGAS scores faithfulness against retrieved context, not ground truth. A 0.95 faithfulness score on stale or incorrect retrieved documents is meaningless. The framework detects retrieval-answer inconsistency; it cannot detect retrieval-reality inconsistency. Every RAGAS deployment requires a parallel data freshness audit.

Secondary limitation: context precision/recall metrics degrade when chunks are long and heterogeneous — the LLM judge struggles to distinguish partial relevance.

Continuous Eval in CI/CD

Three documented patterns cover the enterprise space.

Braintrust eval-action

braintrustdata/eval-action on GitHub Marketplace. Four-step workflow: (1) define eval datasets as input/expected-output pairs, (2) configure scoring functions (LLM rubric, exact match, regex, embedding similarity), (3) add to .github/workflows/, (4) action posts score table as PR comment and optionally fails build on threshold breach.

Key capability: with use_proxy enabled, repetitive LLM judge calls are cached — can cut eval run cost 60–80% on stable datasets. Experiment results versioned with git metadata for regression tracking across fine-tuning runs.

Gap: Braintrust does not natively block merges via required status checks — score table is informational by default unless a custom fail step is added.

DeepEval + pytest

GitHub: github.com/confident-ai/deepeval | Stars: ~14,700 | License: Apache-2.0

The cleanest CI/CD-native option. Plugs into pytest via assert_test(), enabling standard quality gates.

- run: poetry run deepeval test run test_llm_app.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Fails the action if any metric assertion fails. Supports 50+ metrics including multi-turn (ConversationCompleteness, RoleAdherence) and agentic (TaskCompletion, ToolCallAccuracy). [TIER 1 — primary source, official docs]

W&B Weave

Evaluation dashboards and scorer frameworks (exact match, rubric, regex) with time-series leaderboards across fine-tuning runs. Does not provide native CI/CD merge-blocking gates. Teams using W&B typically trigger evals as post-commit Launch jobs, then review dashboards manually. Better suited for fine-tuning iteration tracking than hard deployment gates. [TIER 1 — W&B docs]

What a Working Pipeline Looks Like

The documented enterprise pattern combines tools: W&B for training-time experiment tracking + DeepEval or Braintrust for PR-gating eval + RAGAS for RAG-specific component scoring.

A PR touching the fine-tuned adapter or retrieval pipeline triggers a GitHub Action that: (1) loads a fixed 200-sample golden dataset, (2) runs DeepEval metrics, (3) fails the PR if accuracy drops >2% or faithfulness drops below threshold, (4) posts the Braintrust experiment link for human review of borderline cases.

LLM-as-Judge for Domain Tasks

Prometheus 2 (arXiv:2405.01535)

An open reward model (7B and 8×7B variants) fine-tuned specifically to evaluate other LLMs. Supports both direct assessment (score 1–5 on a rubric) and pairwise ranking. On four direct-assessment benchmarks and four pairwise benchmarks, Prometheus 2 achieves the highest correlation with human judges among all tested open evaluators. [TIER 1 — peer-reviewed, published May 2024, actively maintained on HF Hub]

Rubric structure. Every Prometheus evaluation prompt requires four components: (1) the instruction given to the model being evaluated, (2) the model’s response, (3) a reference answer showing what a score-5 response looks like, (4) a custom 1–5 scoring rubric with a specific descriptor for each score level. Writing quality rubrics is the most time-intensive part. Reference answer is optional for pairwise comparison but required for reliable direct scoring.

When to use local judge vs. external judge:

  • Use Prometheus 2 locally when: the task is narrow and consistent (customer service policy compliance, resolution accuracy), data privacy requirements preclude API calls, or eval volume is high enough that API cost matters.
  • Use GPT-4o or Claude Sonnet as judge when: the task is novel with no training-similar examples, cross-domain comparison is needed, or you are bootstrapping rubrics before enough labeled data exists.

Sample sizes. For directional accuracy/regression: 100–500 examples. For statistical significance on subtle metric differences (p < 0.05, effect size d ≥ 0.2): 500–1,000 examples. [TIER 2 — derived from training data scale guidance]

Practical pattern for Fortune 500 customer-service SLM:

  1. Write 5-level rubrics for each dimension: resolution accuracy, tone appropriateness, policy compliance, conciseness
  2. Create 200 golden examples with reference answers from your best human agents
  3. Run Prometheus 2 (8×7B) locally — eliminates data-leaving-org risk and API cost
  4. Gate on rubric score delta vs. base model
  5. Use GPT-4o as second judge on borderline cases (within 0.5 of threshold)

Domain Eval Benchmarks — Off-the-Shelf Inventory

The landscape has improved but remains patchy.

Finance:

  • AdaptLLM/finance-tasks (HF Hub) — reading comprehension across financial text. [TIER 1]
  • FinWorkBench/Finch — enterprise benchmark for finance and accounting spreadsheet workflows (arXiv:2512.13168, Dec 2024). GPT-5.1 Pro passes only 38.4% of Finch workflows — establishes a usable floor for SLM comparison. [TIER 1 — peer-reviewed]
  • luojunyu/FinMME — multimodal finance reasoning. [TIER 1]

Legal:

  • marvintong/legal-llm-benchmark (HF Hub). [TIER 1]
  • IBM Research 25-dataset enterprise eval suite covering legal, financial services, cybersecurity, climate/sustainability (arXiv:2410.12857, NAACL 2025). [TIER 1 — peer-reviewed]

Customer service:

  • pranav301102/customer-service-for-llm (HF Hub) — community-quality, not production-grade.
  • arXiv:2602.00665 (Feb 2025) provides a benchmark methodology for multi-turn customer service QA but no canonical off-the-shelf eval set has emerged. This is the largest gap in the off-the-shelf landscape.

HR / IT helpdesk: No canonical public benchmark exists. Teams build private golden datasets from ServiceNow/Workday exports and evaluate against human-agent resolution accuracy.

Cross-domain retrieval: RTEB (Retrieval Text Embedding Benchmark, HF Hub) — covers law, healthcare, code, and finance as retrieval tasks. Useful for evaluating the retrieval component of a RAG pipeline independently. [TIER 1]

The Eval-Before-Deploy Checklist

Synthesized from ACL 2025 safety literature, DeepEval documentation, and W&B evaluation guidance.

Accuracy / Task Performance

  • [ ] Golden dataset accuracy vs. base model (minimum 200 samples, domain-specific)
  • [ ] RAGAS faithfulness ≥ threshold if RAG component present
  • [ ] Rubric score on each task dimension (Prometheus 2 or GPT-4o judge)
  • [ ] Edge-case accuracy: adversarial inputs, out-of-scope queries, ambiguous intent

Safety Regression — Most Underestimated Risk

Fine-tuning on benign datasets (GSM8K, Alpaca) partially removes safety guardrails from aligned base models. The mechanism: alignment occupies sparse parameter regions that downstream optimization displaces. Fine-tuning on standard SFT data has been confirmed to increase harmful output rates in multiple 2025 papers. Format-induced degradation is highest for bullet/list/math outputs. [TIER 1 — ACL 2025, arXiv:2605.04572, arXiv:2602.17546]

  • [ ] Run S-Eval or equivalent safety battery against fine-tuned weights vs. base weights
  • [ ] Measure refusal rate on a fixed set of policy-violating prompts (should not drop from base)
  • [ ] Check format-induced degradation: test bullet/list/math outputs specifically

Latency / Cost

  • [ ] P50 / P95 token generation latency on target hardware
  • [ ] Tokens-per-second at expected concurrency
  • [ ] Cost per 1,000 queries vs. base model (LoRA adapters should not increase this)

Regression Against Base Model

  • [ ] Run the same golden set on the base model — fine-tuned model should not regress on general tasks
  • [ ] MMLU or HellaSwag 15-question spot check for catastrophic forgetting (gate, not benchmark)

Behavioral Consistency

  • [ ] Semantic consistency: same input, 5 samples, check answer variance (high variance = fine-tuning instability)
  • [ ] Citation/grounding accuracy: does the model hallucinate sources it was never given?

Human Eval Gate

  • [ ] 50-sample human review by a subject-matter expert before any production deployment. Automated eval catches regressions; human eval catches wrong rubrics.

Decision Table: Eval Tool × Use Case

Tool RAG Pipeline Fine-Tuned SLM Agent / Multi-Turn Base Model Comparison
RAGAS Primary. Faithfulness + context precision/recall purpose-built. Not designed for it. Weak — no multi-turn support. Not applicable.
DeepEval Strong — RAG metrics + CI/CD gate. Second choice to RAGAS for pure RAG. Best CI/CD gate. pytest blocks bad fine-tune PRs. Strong. Multi-turn and agent metrics native. Adequate via G-Eval custom rubrics.
Braintrust Good. Experiment tracking across retrieval parameter sweeps. Good for iteration tracking; weak on hard merge gates. Adequate. Strong for A/B comparison across model versions.
W&B Weave Adequate. Dashboard-based, not gate-based. Strong for training-time regression tracking across checkpoints. Adequate. Best for checkpoint comparison dashboards.
Prometheus 2 Can judge answer quality, no retrieval metrics. Primary for domain rubric scoring. Run locally for data privacy. Can evaluate turn quality with custom rubric. Strong when cost-free, private judge needed.
IBM Enterprise / Finch Not applicable. Secondary validation against public domain baseline. Not applicable. Primary reference for finance/legal domain floor.
S-Eval / safety battery Run as supplement for customer-facing RAG. Mandatory before any fine-tuned model deployment. Mandatory. Baseline establishment only.

Bottom Line

The eval stack is not one tool. It is:

  • RAGAS for RAG component scoring
  • DeepEval for CI/CD gating
  • Prometheus 2 running locally for domain rubric evaluation (eliminates data-leaving-org risk, no API cost)
  • W&B Weave for checkpoint regression dashboards across fine-tuning runs
  • A mandatory safety battery (S-Eval or equivalent) before any fine-tuned weights reach production

The safety regression risk from fine-tuning on benign data is documented and underestimated. Treat it as a first-class eval category, not a checkbox.

Sources

  1. RAGAS GitHub (github.com/explodinggradients/ragas) — ~13,300 stars, Apache-2.0. TIER 1.
  2. RAGAS Cost Documentation (docs.ragas.io/en/stable/howtos/applications/_cost/). TIER 1.
  3. DeepEval GitHub (github.com/confident-ai/deepeval) — ~14,700 stars, Apache-2.0. TIER 1.
  4. DeepEval Unit Testing in CI/CD (deepeval.com/docs/evaluation-unit-testing-in-ci-cd). TIER 1.
  5. Braintrust eval GitHub Action (github.com/marketplace/actions/braintrust-eval). TIER 1.
  6. Prometheus 2 — arXiv:2405.01535, May 2024. TIER 1 (peer-reviewed).
  7. Enterprise Benchmarks for LLM Evaluation — IBM Research / NAACL 2025 (arXiv:2410.12857). TIER 1.
  8. Finch: Finance & Accounting Enterprise Benchmark — arXiv:2512.13168, Dec 2024. TIER 1.
  9. AdaptLLM/finance-tasks — HuggingFace Hub. TIER 1.
  10. marvintong/legal-llm-benchmark — HuggingFace Hub. TIER 1.
  11. Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency — ACL 2025. TIER 1.
  12. Parameter Dynamics and Safety Degradation in LLM Fine-tuning — arXiv:2605.04572. TIER 1.
  13. Learning to Stay Safe: Adaptive Regularization — arXiv:2602.17546. TIER 1.
  14. W&B Weave Evaluations (wandb.ai/site/evaluations/). TIER 1.
  15. Can SLMs Handle Multi-Turn Customer-Service QA? — arXiv:2602.00665, Feb 2025. TIER 1.

Brandon Sneider | brandon@brandonsneider.com May 2026