Source files: sources/21-benchmarks/eval-authoring-frameworks-2026-raw.md.
Source status: primary documentation and research-paper ledger, retrieved 2026-07-22.
| Framework | Generation unit | Runtime/verifier strength | Best use |
|---|---|---|---|
| LangChain eval-engineering + Harbor | repository/trace-informed capability and Harbor task | strongest environment, trajectory, reward, and verifier boundary | coding and tool-using agents |
| Google ADK | agent-definition conversation scenarios and recorded conformance cases | replay and agent/tool interaction; live eval support is uneven by command | Google agents and A2A/ADK conformance |
| Microsoft Agent Framework | EvalItem conversations plus provider evaluators |
provider-agnostic evaluator orchestration; Foundry adds managed evaluators | Microsoft/.NET/Python enterprise agents |
| Pydantic Evals | typed cases, datasets, expected outputs, evaluators | strong deterministic contract checking; no sandbox harness | structured outputs, tool args, state/artifact contracts |
| W&B Weave | datasets, tasks, scorers, trace-backed examples | strong experiment/trace analysis; environment is application-owned | model, pipeline, and production feedback loops |
| Copilot Studio | simulated real-world agent tests | managed product-level test execution | Microsoft Copilot deployments |
| DeepEval/Ragas | synthetic goldens and RAG testsets | metric/evaluator layer; environment is application-owned | RAG and response-quality coverage |
| SkillGenBench/SkillGen | generated skill artifacts and skill-generation measurements | research benchmark or verified synthesis, not a production runner | evaluating skill-generation systems |
Pydantic boundary
Pydantic Evals is the typed contract layer. In the local GitLab architecture it should sit inside the verifier or adapter: validate the agent’s output model, tool arguments, required fields, state transition, and artifact metadata before semantic scoring. It should not be treated as a replacement for Harbor’s environment isolation or for trace/dataset systems.
Verification rule
Generation is not acceptance. Every generated case requires provenance, an independently reviewed oracle, positive and negative controls, hidden expected material, and a failure-domain label. Recorded interactions are replay evidence, not automatically correct answers.
Evidence Boundaries
Google ADK, Microsoft Agent Framework, Pydantic, Weave, and Copilot Studio are product/documentation evidence. SkillGenBench and SkillGen are research evidence. Exact export fields, provider-specific evaluator behavior, and cloud-service pricing remain version-sensitive and need package-pinned fixtures before schema claims are promoted to production contracts.