framework: DeepEval

gap_text: Package-pinned local no-key TestRun JSON is captured for deepeval==4.0.6

status: captured

verification_needed: Keep native_deepeval_test_run_real_run_001 validating with raw export hash.

owner_role: eval-platform

notes: Captured evidence row retained so the ledger can mix open gaps with verified package fixtures without ambiguity.


gap_text: Hosted Confident AI export shape still needs capture

status: open

verification_needed: Run or obtain a Confident AI hosted project export from a pinned DeepEval/Confident AI version and store raw payload plus hash.

notes: Keep separate from local TestRun JSON because hosted project exports can include dashboard-only fields.


gap_text: Component trace export shape still needs capture

verification_needed: Capture a component trace export with retrieval/tool/evaluator spans and map raw span fields to trace_ref_v1.

notes: Needed before claiming DeepEval coverage for agentic chat-cycle failures.


gap_text: Built-in LLM judge metrics still need pinned fixtures with model/provider metadata

verification_needed: Run a built-in DeepEval LLM metric with explicit judge model/provider metadata and store the local TestRun JSON.

notes: The current fixture uses a deterministic custom BaseMetric and intentionally avoids API keys.


framework: Hugging Face Evaluate

gap_text: Package-pinned evaluate.save serialization is captured for evaluate==0.4.6

verification_needed: Keep native_huggingface_evaluate_save_real_run_001 validating with raw export hash.

notes: Captured serialization only; metric execution remains separate.


gap_text: evaluate.load metric-module execution still needs selected per-metric pinned fixtures

verification_needed: Run selected evaluate.load metrics offline or with pinned Hub revisions and store output dictionaries plus runtime metadata.

owner_role: ml-platform

notes: Needed for per-metric adapters; Evaluate intentionally lacks one universal output schema.


gap_text: EvaluationSuite and distributed cache artifacts still need pinned captures

verification_needed: Run EvaluationSuite and distributed/cache paths with pinned versions and store emitted artifacts.

notes: Relevant for model benchmark reproducibility and shared runners.


framework: RAGAS

gap_text: Package-pinned dataset JSONL serialization is captured for ragas==0.4.3

verification_needed: Keep native_ragas_dataset_jsonl_real_run_001 validating with raw export hash.

notes: Covers sample serialization, not scored results.


gap_text: Scored EvaluationResult dataframe/JSON shape still needs a pinned run with valid ragas_traces

verification_needed: Run RAGAS scoring with a deterministic or pinned judge/embedding setup and capture EvaluationResult, dataframe/JSON, cost, traces, and ragas_traces fields.

notes: Required before treating RAGAS metric rows as production adapter evidence.


gap_text: Metric-specific required-field matrix still needs capture for selected enterprise RAG metrics

verification_needed: For selected RAGAS metrics, record required sample fields, optional fields, failure behavior, and normalization mapping.

notes: Needed to prevent adapters from running metrics against incomplete sample rows.


framework: Promptfoo

gap_text: No standalone published JSON Schema for exports

verification_needed: Continue pinning Promptfoo package versions and validating raw JSON/JSONL exports because no standalone upstream JSON Schema is available.

notes: This is a permanent adapter posture unless Promptfoo publishes stable schemas.


gap_text: Separate red-team report UI/export schema needs deeper source tracing

verification_needed: Capture Promptfoo red-team report exports or UI data payloads and map plugin/strategy dimensions to failure taxonomy.

owner_role: security-eval

notes: Normal eval JSON/JSONL is pinned; red-team reporting remains separate.


gap_text: Version-pinned fixture needed before production adapter lock

verification_needed: Promptfoo 0.121.17 JSON and JSONL echo-provider fixtures are captured; add provider-specific variants before locking provider adapters.

notes: Captured for generic export shape, still limited for provider-specific response bodies.


framework: OpenEvals

gap_text: TypeScript package object types need the same source-level pass

verification_needed: Inspect OpenEvals TypeScript source and capture evaluator result shapes and custom-output behavior.

notes: Python object shape is captured; TS package may differ.


gap_text: LangSmith docs do not expose a full serialized feedback/run payload for OpenEvals

verification_needed: Capture OpenEvals-to-LangSmith feedback serialization from a real run or official export.

notes: Needed when OpenEvals is used as a LangSmith feedback writer rather than a standalone result object.


gap_text: Upstream intentionally has no universal metric result schema

verification_needed: Treat each selected metric as a separate adapter contract with fixture-backed key mapping.

notes: This duplicate framework section covers metric-boundary posture, not evaluate.save serialization.


gap_text: Enterprise adapters need per-metric mappings for selected metrics

verification_needed: Declare metric-specific mappings for accuracy, exact match, F1, toxicity/safety, and domain-specific metrics selected for enterprise use.

notes: Avoids treating arbitrary Evaluate metric dictionaries as normalized eval_result_v1 rows.


framework: Hugging Face LightEval

gap_text: Need full LightEval CLI run over a real model/dataset to confirm task config population and backend-specific export variations

verification_needed: Run LightEval CLI with a pinned tiny model/dataset or controlled local task and capture result JSON, detail parquet, config_tasks, and backend metadata.

notes: Logger-level fixture is useful but does not exercise full task loading or model inference.


gap_text: Logger-level lighteval==0.13.0 fixture confirms result JSON sections and detail parquet columns doc, metric, and model_response

verification_needed: Keep native_lighteval_logger_bundle_real_run_001 validating with raw export hash.

notes: Captured by EvaluationTracker.save, not by CLI.


framework: OpenAI Agents SDK

gap_text: Hosted tracing unavailable for OpenAI API ZDR organizations

verification_needed: Document custom trace processor requirements and capture raw processor export fixtures for ZDR-compatible deployments.

owner_role: agent-platform

notes: Enterprise adapters cannot assume hosted trace visibility.


gap_text: Production adapter should capture raw processor/export payloads from a concrete trace processor

verification_needed: Run Agents SDK with a custom trace processor and store raw generation/tool/guardrail/handoff span payloads.

notes: Needed for trace_ref_v1 adapter proof.


framework: Google ADK

gap_text: Google Agent Engine platform event export shape still needs capture beyond open-source ADK

verification_needed: Capture Google Agent Engine event/export payloads and compare against open-source ADK Event and EvalSet objects.

notes: Open-source ADK fields are captured; hosted platform exports may differ.


framework: Microsoft AutoGen

gap_text: Microsoft Foundry-specific eval and telemetry exports were not verified in this pass

verification_needed: Capture Foundry eval and telemetry export payloads and map any AutoGen/OpenTelemetry overlap.

notes: AutoGen OTel runtime source is covered; Foundry platform export remains separate.


framework: Mistral Agents and Conversations

source_ledger_ref: sources/21-benchmarks/eval-native-schema-boundaries-2026.json#Mistral Agents and Conversations.remaining_gaps[0]

gap_text: Mistral observability judge/eval object schemas need OpenAPI-level extraction

verification_needed: Extract Mistral OpenAPI objects for observability, judge, and eval payloads and add source-observed fixtures.

notes: Conversation output entries are covered; eval/judge objects are not.


framework: Qwen-Agent

gap_text: No first-class Qwen-Agent trace/event telemetry envelope found in primary repo files

verification_needed: Confirm whether Qwen-Agent has a durable trace/event envelope or define a wrapper adapter around message and ReAct stream records.

notes: Message/function-call stream is covered; first-class telemetry was not found.