framework: Promptfoo

native_grain: per-result EvaluateResult row

description: HR policy citation case

question: Can a US employee carry over unused PTO into next year?

type: contains-json

type: llm-rubric

value: Answer only from retrieved HR policy context.

prompt: Answer the employee policy question using only the retrieved snippets.

question: Can a US employee carry over unused PTO into next year?

output: Employees may carry over up to 40 hours of unused PTO, subject to local policy.

reason: The answer is grounded and includes the required carryover limit.

pluginId: policy-grounding

required_native_fields: id testIdx testCase provider response success score gradingResult evaluationId traceId

optional_native_fields: namedScores cost tokenUsage metadata latencyMs

normalization_notes: Use evaluationId as run lineage, id as native result key, traceId as trace_ref candidate, score/gradingResult as metric evidence, and preserve full row behind harness.native_export_ref.

known_limitations: Source-observed interface fixture, not a CLI-generated run artifact.

known_limitations: Promptfoo has no standalone versioned JSON Schema for this export.


framework: OpenEvals

native_grain: EvaluatorResult return object

key: correctness

comment: The answer is directionally correct but omits the policy exception for union employees.

__ls_framework: openevals

__ls_language: python

required_native_fields: key score

optional_native_fields: comment metadata source_run_id

normalization_notes: Map key to metric id, score to normalized metric score, comment to reason_ref or inline rationale artifact, metadata to evaluator metadata, and source_run_id to trace/run foreign key.

known_limitations: LangSmith feedback/run serialization must be captured separately when OpenEvals logs into LangSmith.

known_limitations: TypeScript package output shape still needs a parallel pinned fixture.


framework: Hugging Face LightEval

native_grain: aggregate result JSON plus detail artifact contract

backend: transformers

query: What is the PTO carryover limit?

input: What is the PTO carryover limit?

text: 40 hours

text_post_processed: 40 hours

normalization_notes: Normalize aggregate metrics into model-candidate eval results and retain detail Parquet/JSON paths as artifacts. Treat sample detail rows as evidence only when linked to a repo eval case.

known_limitations: This fixture reflects documented/source-observed layout, not a package-generated artifact.

known_limitations: The separate native_lighteval_logger_bundle_real_run_001 fixture confirms logger-level detail Parquet columns doc, metric, and model_response.

known_limitations: A full CLI run over a real model/dataset is still needed to confirm populated config_tasks and backend-specific export variations.


framework: DeepEval

native_grain: LLMTestCase plus tool and retrieval context fields

input: Can a US employee carry over unused PTO into next year?

actual_output: Employees may carry over up to 40 hours of unused PTO.

expected_output: US employees may carry over up to 40 hours unless a local policy or bargaining agreement says otherwise.

context: HR PTO policy section 4.2 states the 40 hour carryover rule.

retrieval_context: Confluence HR PTO section 4.2: Employees may carry over up to 40 hours.

kb.search

description: Searches approved HR policy knowledge base

reasoning: Policy answer requires current HR source text.

output: Confluence HR PTO section 4.2

query: PTO carryover US employees

query: PTO carryover US employees

normalization_notes: Map input/expected_output to case fields, actual_output to attempt artifact, context to reference evidence, retrieval_context to retrieved evidence, and tools_called/expected_tools to tool-action failure evidence.

known_limitations: This fixture pins the test-case object, not DeepEval's local JSON report/export schema.


framework: RAGAS

native_grain: SingleTurnSample plus EvaluationResult boundary

user_input: Can a US employee carry over unused PTO into next year?

retrieved_contexts: Confluence HR PTO section 4.2: Employees may carry over up to 40 hours.

reference_contexts: HR PTO policy section 4.2 states the 40 hour carryover rule and local exceptions.

response: Employees may carry over up to 40 hours of unused PTO.

reference: US employees may carry over up to 40 hours unless local policy or bargaining agreement differs.

faithfulness: Answer must be supported by retrieved contexts.

citation_span_accuracy: Cited span must contain the carryover rule.

query_length: short

traces: provider-dependent

normalization_notes: Keep retrieved_contexts and reference_contexts separate. Use sample fields for case/evidence construction and EvaluationResult scores only as metric output evidence.

known_limitations: Exact serialized dataframe/JSON result shape and metric-required-field matrix still require real package-pinned fixtures.


framework: LangSmith

native_grain: run/span data format and dataset example link

run_type: chain

question: Can a US employee carry over unused PTO into next year?

answer: Employees may carry over up to 40 hours of unused PTO.

required_native_fields: id trace_id inputs outputs

normalization_notes: Use LangSmith run IDs as foreign keys. Preserve run tree as trace evidence; map feedback_stats only after evaluator key and source feedback object are captured.

known_limitations: Full serialized feedback object, annotation queue export, and experiment export edge cases still need pinned export fixtures.


framework: Phoenix / OpenInference

native_grain: OpenTelemetry/OpenInference trace span plus eval output boundary

llm.generate

code: OK

openinference.span.kind: LLM

input.value: Can a US employee carry over unused PTO into next year?

output.value: Employees may carry over up to 40 hours of unused PTO.

service.name: hr-policy-agent

label: correct

explanation: The answer is supported by the retrieved PTO policy span.

normalization_notes: Normalize the trace as trace_ref_v1 with OpenTelemetry/OpenInference semantic convention metadata. Store eval output as score evidence only when tied to an evaluator config and dataset row.

known_limitations: Phoenix dataset, experiment, annotation, and SDK eval result export JSON still need pinned capture.


framework: StateBench contract validator

native_grain: validator CLI result for native fixture contract

command: scripts/.venv-tts/bin/python3 -m statebench.runners.contract_validator --schema native_eval_fixture_v1 --input statebench/fixtures/enterprise-eval-contract/v1/native-fixtures.pass.jsonl --jsonl

stdout: ok — statebench/fixtures/enterprise-eval-contract/v1/native-fixtures.pass.jsonl conforms to native_eval_fixture_v1

required_native_fields: command exit_code stdout runner schema input

normalization_notes: This is a repo-native real-run export proving the validator accepted the native fixture file. It is not evidence that external vendor CLIs produced their payloads; vendor package real-run fixtures remain separate work.

known_limitations: Covers StateBench validator behavior only.

known_limitations: External framework package-pinned exports still need capture when dependencies are available.


native_grain: CLI JSON export from promptfoo eval

command: npx --yes promptfoo@0.121.17 eval --config promptfooconfig.yaml --output /private/tmp/stateofai-promptfoo-realrun/promptfoo-realrun.json --no-progress-bar

package: promptfoo

provider: echo

native_top_level_fields: evalId results config shareableUrl metadata vars runtimeOptions

optional_native_fields: shareableUrl vars runtimeOptions results.prompts

normalization_notes: This is an external package-pinned real-run JSON export. Use results.results rows as native Promptfoo EvaluateResult evidence and retain the whole JSON export behind harness.native_export_ref.

known_limitations: Uses the local echo provider, so it validates export shape without exercising remote model providers.

known_limitations: Does not cover Promptfoo red-team report exports or JSONL output.


native_grain: CLI JSONL per-result export from promptfoo eval

command: npx --yes promptfoo@0.121.17 eval --config promptfooconfig.yaml --output /private/tmp/stateofai-promptfoo-realrun/promptfoo-realrun.jsonl --no-progress-bar

native_top_level_fields: cost gradingResult id latencyMs prompt promptId promptIdx provider response score success testCase testIdx tokenUsage vars

required_native_fields: id testIdx testCase provider response success score gradingResult

optional_native_fields: cost latencyMs namedScores promptId tokenUsage vars metadata failureReason

normalization_notes: This is an external package-pinned real-run JSONL export. Treat each JSONL row as one Promptfoo EvaluateResult-style native result row and retain the raw JSONL export behind harness.native_export_ref.

known_limitations: Uses the local echo provider, so it validates per-result export shape without exercising remote model providers.

known_limitations: Does not cover Promptfoo red-team report exports.


native_grain: EvaluationDataset.to_jsonl output for a SingleTurnSample

package: ragas

normalization_notes: This is an external package-pinned RAGAS dataset JSONL serialization fixture. Treat the JSONL row as sample/case evidence; do not treat it as a scored EvaluationResult.

known_limitations: Covers EvaluationDataset.to_jsonl and SingleTurnSample serialization only.

known_limitations: Does not cover EvaluationResult scores, cost callback, traces, ragas_traces, or metric-specific required fields.

known_limitations: Top-level ragas import in this isolated install attempted an optional VertexAI path; this fixture loads ragas.dataset_schema directly from the installed package to exercise the serialization class without model or provider dependencies.


framework: Hugging Face Evaluate

native_grain: evaluate.save JSON result artifact

metric_name: accuracy

package: evaluate

normalization_notes: This is an external package-pinned Hugging Face Evaluate serialization fixture. Treat the JSON as a metric result artifact with Evaluate runtime metadata; normalize metric-specific keys only when a per-metric adapter declares them.

known_limitations: Covers evaluate.save serialization only, not evaluate.load metric-module execution.

known_limitations: Accuracy was supplied as a deterministic offline result to avoid Hugging Face Hub downloads.

known_limitations: Does not exercise model, dataset, distributed cache, or EvaluationSuite execution paths.


native_grain: offline deepeval.evaluate TestRun JSON

metric_data_fields: name threshold success score reason strictMode evaluationModel evaluationCost verboseLogs

metric_name: Exact Contains

native_top_level_fields: testCases conversationalTestCases metricsScores identifier prompts testPassed testFailed runDuration evaluationCost official

package: deepeval

serialization_api: deepeval.evaluate local TestRun JSON

test_case_fields: name input actualOutput expectedOutput context retrievalContext tags success metricsData runDuration evaluationCost order

required_native_fields: testCases metricsScores testPassed testFailed identifier

optional_native_fields: conversationalTestCases prompts runDuration evaluationCost official

normalization_notes: This is an external package-pinned DeepEval offline TestRun JSON generated with a deterministic custom BaseMetric. Map testCases rows to eval attempts and metricsData/metricsScores to normalized metric scores while retaining the raw TestRun artifact.

known_limitations: Covers local deepeval.evaluate TestRun serialization only, not Confident AI hosted exports or component trace export shapes.

known_limitations: Uses a deterministic custom BaseMetric to avoid API keys; built-in LLM metrics and judge models still need separate pinned fixtures.

known_limitations: DeepEval attempted PostHog telemetry after the successful run and DNS failed, but the process exit code remained 0.

known_limitations: Run was executed from /private/tmp to avoid committing DeepEval's .deepeval/.latest_run_full.json side effect.


native_grain: EvaluationTracker.save aggregate JSON plus detail parquet bundle manifest

detail_parquet_fields: doc metric model_response

package: lighteval

normalization_notes: This is an external package-pinned LightEval logger-level bundle fixture generated through EvaluationTracker.save. Normalize aggregate metrics from results and retain the detail parquet bundle as sample evidence when linked to a repo eval case.

known_limitations: Covers LightEval EvaluationTracker.save serialization and observed detail parquet columns only.

known_limitations: Does not run the full LightEval CLI over a downloaded model or dataset.

known_limitations: config_tasks is empty in this logger-level fixture because no LightevalTask object was loaded.

known_limitations: LightEval was installed into an isolated target directory; pip reported dependency conflicts against the ambient environment, but imports and logger save completed under PYTHONPATH.