Frontier Lab Evaluation Research and SDK Primitive Crosswalk — 2026

Source files: sources/21-benchmarks/frontier-lab-eval-sdk-crosswalk-2026-raw.md · sources/21-benchmarks/frontier-lab-eval-sdk-primary-sources-2026.md · sources/21-benchmarks/frontier-lab-eval-sdk-primary-sources-2026.jsonl · sources/21-benchmarks/frontier-lab-eval-type-matrix-2026.json · sources/21-benchmarks/frontier-lab-eval-sdk-primary-sources/raw/ · research/21-benchmarks/raw/frontier-lab-eval-sdk-*.pdf · research/21-benchmarks/raw/frontier-lab-eval-sdk-*.chandra.txt · research/21-benchmarks/enterprise-evaluation-knowledge-coding-agents-2026.md · research/21-benchmarks/decentralized-gitlab-eval-federation-2026.md · docs/research-synthesis-verification-loop.md

See also: Enterprise Evaluation for Knowledge Bases and Coding Agents · Decentralized GitLab Eval Federation · Benchmark Foundations · Agent Observability Format Standard

Source status: This report synthesizes primary lab documentation, SDK references, model-card benchmark disclosures, and eval papers retrieved on June 21, 2026. The primary-source ingestion inventory captures 43 source rows with local raw mirrors or fallback raw repository/current-doc captures; PDF-backed papers are stored under research/21-benchmarks/raw/ with Chandra text extraction where available. It distinguishes implementation primitives from measured effectiveness. Vendor/lab docs are treated as capability evidence; model cards are treated as benchmark disclosures; papers are treated as method evidence. None of these replace private enterprise eval packs.

Executive Summary

Frontier labs are converging on the same evaluation thesis: the evaluated unit is no longer only the model; it is the model plus tools, context, policies, traces, and task state. The SDK primitive layer is becoming as important as benchmark scores.

The overlap is clear:

• Task datasets and private golden sets appear in OpenAI Evals, Anthropic Console evals, Google Gen AI Evaluation, LangSmith/DeepEval/RAGAS-style workflows, and enterprise benchmark-pack practice.
• Rubrics and LLM judges appear in OpenAI graders, Anthropic prompt-quality grading, Google adaptive/rubric metrics, ADK final-response metrics, DeepMind FACTS judges, and TencentLLMEval human/LLM evaluation processes.
• Trajectory and tool-use evaluation appears in Google ADK, OpenAI Agents traces, Anthropic Agent SDK hooks/subagents/MCP, Microsoft AutoGen teams, Mistral Agents and function calling, Qwen-Agent, and Bedrock Agents.
• Groundedness and knowledge-base evaluation appears in Google FACTS Grounding/SAFE, Bedrock Knowledge Bases, Mistral citations/document libraries, Cohere search/rerank, RAGAS, and enterprise KB eval stacks.
• Interoperability evaluation is emerging from A2A and MCP. These are not eval frameworks, but they define observable contracts: capability discovery, authorization, task lifecycle, resources, tools, sampling, artifacts, and consent.

The divergence is equally important:

• OpenAI, Google, Anthropic, Microsoft, AWS, Mistral, and Meta Llama Stack expose eval or agent/runtime primitives that can be instrumented directly. Meta is stronger on eval/scoring/RAG benchmark infrastructure than a model-card-only reading would imply, but still less explicit on app-agent trajectory evaluation than Google ADK or Anthropic/OpenAI agent SDKs.
• DeepSeek, Moonshot/Kimi, Z.ai, Tencent, ByteDance Seed, Baidu, InternLM, 01.AI, Baichuan, StepFun, and MiniMax often expose model cards, benchmarks, OpenAI-compatible APIs, or deployment toolkits, but less first-party eval operating-system infrastructure. They should be evaluated through the enterprise harness rather than trusted through vendor-native eval surfaces.
• Qwen is the main Chinese open-weight exception because Qwen-Agent and Qwen function-calling docs provide a concrete local agent/tool/RAG harness surface.
• NVIDIA is not a frontier model lab in the same sense here; its evaluation contribution is the security/guardrail lane through garak and NeMo Guardrails.

The operating recommendation is to standardize the enterprise eval result contract, not the lab SDK. Each lab SDK can run locally in a repo or platform environment, but every run should emit the same normalized result row: model, harness, SDK primitive, task, trace, source bundle, judge, score, failure code, and artifact references.

Cross-Lab Concept Matrix

Post-Ingestion Delta

The deeper ingestion does not change the core finding: enterprises should standardize result records and source/trace contracts rather than standardizing on one lab SDK. It does change four details:

1. Meta is stronger than the first-pass synthesis implied. The Llama Stack repository includes explicit eval references, scoring providers, CLI eval templates, and RAG benchmark examples. Meta should sit in the “self-hosted eval substrate” lane, not only the “model-card disclosure” lane.
2. Baidu is stronger on current open-tooling than the older ERNIE 3.0 paper alone showed. The ERNIE repository now exposes ERNIE 4.5, ERNIEKit, FastDeploy, function-call training support, and a knowledge-retrieval cookbook. That strengthens Baidu as a Chinese KB/function-call/model-toolkit lane, while still not making it a first-party eval operating system.
3. InternLM coverage is weaker than the first-pass wording implied. The rendered ReadTheDocs source was rate-limited and the fallback artifact is the InternLM repository README, not a full LMDeploy/agent/eval docs ingestion. Keep InternLM as an external harness lane until LMDeploy-specific sources are captured.
4. MiniMax should be treated as a moved-docs/API-model lane, not an eval lane. The old /platform/document URL 404s; the current docs URL captured a model overview page and navigation that includes MCP, but not a first-class eval primitive.

Visual Readout

The matrix below is a capability-evidence map, not a leaderboard. Scores are derived from the ingested primary-source corpus:

• 0 means no relevant support was captured in the ingested public sources.
• 1 means candidate evidence, model-card evidence, or benchmark disclosure.
• 2 means an instrumentable runtime, SDK, API, or toolkit primitive.
• 3 means an explicit eval framework, eval service, benchmark method, or first-party evaluation workflow.

The Teach-informed reading order is: mission, reference, practice, retention. The mission is not to memorize vendor names. The mission is to classify a failure quickly enough that the right owner can fix it and the right artifact can preserve it. Every visual should therefore answer five questions before it compares labs:

1. Where in the agentic chat cycle did the evidence appear?
2. Which artifact failed: fixture, retrieval context, tool call, final answer, judge record, trace, or promotion record?
3. Which failure code should the user apply?
4. What raw evidence proves the classification?
5. Where should the result live: repo-local pack, team rollup, firm warehouse, KB article, or training lineage queue?

This also keeps the Tufte bar high: direct labels beat legends, source/date/provenance must be visible, and the chart has to carry a real classification task rather than merely decorating the report.

The most useful teaching frame is not “which lab wins.” It is where the eval lives:

• Case setup: task fixtures, personas, expected behavior, and source bundles live before the first model call.
• Context retrieval: retrieval traces, chunks, citations, ACL decisions, and freshness checks live between user intent and model context.
• Tool/action path: tool calls, arguments, results, handoffs, MCP/A2A messages, and side effects live in the trajectory.
• Final answer: structured output, citations, refusal behavior, and user-facing text live at response time.
• Judge/check layer: rubric judges, deterministic checks, calibration records, and security probes live after the run.
• Evidence store: eval_result_v1, JUnit, trace references, model versions, and harness versions live in CI artifacts and the eval warehouse.
• Promotion loop: only reviewed failures, benchmark updates, KB entries, and skill changes should be promoted back into durable knowledge.

The broadest evidence category is still model-card benchmark disclosure. That should not be mistaken for deployable enterprise evidence. The operational center of gravity is moving toward trajectory/tool evaluation, RAG and KB grounding, interop contracts, and trace observability because those are the parts that make an agent system auditable.

Eval Type Deep Dive

Lab × Eval-Type Matrix

This table compresses the SVG heatmap into reviewable source text. It should be read with the 0-3 support scale above.

Concept Overlap

Where Labs Diverge

1. Evals as Product API vs Evals as Research Disclosure

OpenAI and Google expose eval as a product/API workflow. Anthropic exposes prompt evals and agent SDK primitives that make evals possible. Microsoft, AWS, and Mistral expose agent runtime surfaces that can be instrumented.

Meta, DeepSeek, Kimi, Z.ai, Tencent, ByteDance, Baidu, 01.AI, Baichuan, StepFun, and MiniMax more often publish model-card benchmark evidence or API access. That is useful for candidate discovery but weaker for enterprise eval operations. These models should enter the same GitLab/StateBench harness as external candidates.

2. Final Answer Scoring vs Trajectory Scoring

Google ADK is the clearest explicit source for agent trajectory evaluation: tool path and final response are separate eval targets. OpenAI traces, Anthropic hooks/subagents, AutoGen teams, Bedrock traces, Mistral handoffs, and Qwen-Agent make similar information observable, even when the docs do not call it an eval taxonomy.

Enterprise implication: every coding-agent and KB-agent run should persist both:

• final_response_scores: correctness, usefulness, style, policy, citation quality;
• trajectory_scores: tool selection, permission boundary, retrieval path, handoff path, retry behavior, side effects, stop reason.

3. Grounding vs Retrieval vs Truth

Google FACTS and SAFE make the most important distinction: factuality decomposes into parametric knowledge, search-using factuality, long-document grounding, and multimodal grounding. RAGAS-style metrics add retrieval/generation decomposition. Bedrock KB, Mistral document library, and Cohere retrieval/reranking provide runtime surfaces.

Enterprise implication: do not score “RAG accuracy” as one number. Separate:

• source retrieval;
• source authority;
• source freshness;
• citation span validity;
• claim support;
• answer utility;
• abstention when sources are insufficient.

4. Standards Are Not Evals

MCP and A2A are standards for connecting systems, not grading systems. Their value for evals is that they create observable contracts. MCP exposes resources, prompts, tools, sampling, roots, elicitation, logging, cancellation, and security/consent boundaries. A2A exposes Agent Cards, tasks, messages, parts, artifacts, and lifecycle.

Enterprise implication: create protocol conformance eval packs:

• mcp_resource_acl_v1;
• mcp_tool_authorization_v1;
• mcp_prompt_injection_boundary_v1;
• a2a_agent_card_capability_truthfulness_v1;
• a2a_task_lifecycle_v1;
• a2a_artifact_integrity_v1.

5. Chinese Open-Weight Labs Need Harness-First Evaluation

Qwen, DeepSeek, Kimi, Z.ai, Tencent, ByteDance, Baidu, InternLM, 01.AI, Baichuan, StepFun, and MiniMax should not be grouped only as “Chinese labs.” They diverge materially:

• Qwen has the strongest local framework surface through Qwen-Agent and Qwen function calling.
• DeepSeek and Kimi publish strong reasoning/agentic benchmark claims and expose OpenAI-compatible APIs, but evals should be external.
• Tencent contributes a useful hierarchical human-alignment eval methodology through TencentLLMEval.
• ByteDance Seed frames real-world agency around search, code execution, and GUI interaction; this points to OSWorld/WebArena/SWE-style harnesses.
• Baidu’s ERNIE lineage is knowledge-enhanced, making it relevant for Chinese KB and knowledge graph tasks, but modern eval/runtime primitives need deeper Qianfan/Wenxin ingestion.
• Z.ai/GLM, InternLM, 01.AI, Baichuan, StepFun, and MiniMax remain external model lanes unless their platform-specific eval docs are captured.

Required Canonical Result Fields

To make lab SDKs comparable, the enterprise result contract should add these fields to eval_result_v1:

{
  "lab": "openai|anthropic|google|meta|microsoft|aws|nvidia|mistral|cohere|qwen|deepseek|kimi|zai|tencent|bytedance|baidu|internlm|yi|baichuan|stepfun|minimax|other",
  "sdk_surface": "eval_api|agent_sdk|trace_api|mcp|a2a|function_calling|rag_kb|model_card_only|openai_compatible_api|local_harness",
  "model_id": "string",
  "model_version": "string",
  "harness_id": "string",
  "harness_version": "string",
  "eval_pack_id": "string",
  "eval_pack_version": "string",
  "task_family": "kb_qa|coding_agent|tool_calling|multi_agent|rag|security|gui|research_synthesis|domain_specific",
  "trajectory_required": true,
  "trace_ref": "string",
  "source_bundle_ref": "string",
  "interoperability_protocol": "none|mcp|a2a|openapi|json_schema|custom",
  "judge_model": "string",
  "judge_rubric_version": "string",
  "deterministic_checks": ["schema", "citation_span", "acl", "unit_tests"],
  "failure_codes": ["retrieval_miss", "bad_tool_arg", "unsupported_claim"]
}

This keeps SDKs decentralized while making team and firm-wide dashboards comparable.

Implementation Implications For The GitLab Eval Federation

1. Do not pick one lab SDK as the enterprise eval layer. Let each repo run the SDK it actually uses.
2. Require adapters. Each SDK should emit the canonical JSONL result row, JUnit summary, and trace artifact references.
3. Separate eval-pack families. Use different packs for OpenAI Evals-style task regression, ADK-style trajectory evals, MCP/A2A conformance, RAG groundedness, and security probing.
4. Treat model-card-only labs as candidates. They enter the same harness through OpenAI-compatible APIs or local inference, but their lab-published scores do not become deployment evidence.
5. Promote only reviewed learnings to KBs. Raw traces stay in trace storage; only redacted, reviewed failure patterns and benchmark-pack guidance enter knowledge bases.

Evidence Boundaries

• Public benchmark claims do not establish enterprise reliability.
• SDK support for tracing does not mean traces are complete enough for governance; check span coverage.
• LLM judges need calibration, judge-version tracking, and disagreement monitoring.
• OpenAI-compatible APIs normalize call shape, not model behavior.
• MCP/A2A conformance does not imply answer quality.
• Chinese open-weight model claims are useful for candidate discovery, but private eval packs should control for censorship, refusal behavior, tool safety, language/domain coverage, and serving-template differences.

Bottom Line

The lab landscape is not moving toward one universal eval tool. It is moving toward observable agent runtimes plus private task-specific evaluation contracts. The enterprise should standardize result records, failure codes, source manifests, trace references, and benchmark-pack governance. Lab SDKs remain interchangeable execution surfaces.