← Benchmarks 🕐 5 min read
Benchmarks

Domain YAML Generation Optimization 2026

Brandon Sneider · June 2026

For domain-specific YAML generation, the first optimization target should not

Executive Summary

For domain-specific YAML generation, the first optimization target should not be fine-tuning. The winning path is usually schema plus deterministic validator, adversarial eval set, prompt optimization with validator feedback, trace mining for real failure cases, and fine-tuning only when prompt optimization and constrained decoding cannot hit the reliability, latency, or cost target.

YAML generation is a good fit for trace-derived optimization because failures are visible. Parse errors, schema violations, enum mistakes, missing required fields, invalid references, and downstream execution failures can all become textual feedback for DSPy/GEPA-style optimizers or evaluation samples for Bedrock Advanced Prompt Optimization.

Format Contracts

Use case Format Record unit Notes
Bedrock Advanced Prompt Optimization JSONL One prompt template per line Requires fixed version, templateId, promptTemplate, and evaluationSamples; supports 1-100 evaluation samples per template
Bedrock customization/fine-tuning JSONL Model-specific example Formats vary: prompt/completion, Converse-style messages, Claude messages; minimum record counts vary by model and method
lm-evaluation-harness YAML task config plus dataset Eval task YAML defines dataset, prompt rendering, metric, few-shot behavior, filters, and output processing
DSPy/GEPA Python program, examples, metric, textual feedback Program optimization trace Optimizes instructions/program text using structured execution traces and feedback
Local model SFT Usually JSONL or chat-template-formatted records Training example Exact format depends on trainer/model tokenizer; chat template mismatch can break structured output behavior
Trace-derived corpus Raw events plus normalized examples Episode, span, or final output Needs redaction, provenance, labels, and train/eval split before tuning
Tier Example count Purpose Exit criterion
Smoke eval 10-20 Find obvious parse/schema failures and prompt misunderstandings Baseline prompt can produce valid YAML on easy cases
Prompt optimization 30-100 Optimize instructions against known variants and validator feedback Stable pass rate on held-out cases, not just training samples
Coverage eval 100-300 Cover field combinations, edge cases, aliases, missing data, and dangerous defaults Failure modes become rare and explainable
SFT candidate 300-1,000+ Teach persistent house style, domain shorthand, or repeated transformation patterns Prompt+validator loop fails target or is too costly/slow

The numbers are operating bands, not universal laws. The quality and coverage of the examples dominate the raw count. Fifty examples that cover every branch of a YAML schema are more valuable than five hundred near-duplicates.

YAML Failure Taxonomy

Failure type Example signal Best first fix
Parse error Invalid indentation, unclosed string, illegal scalar Constrained decoding or stricter output wrapper
Schema error Missing required field, wrong type, unknown key Schema-aware prompt plus validator feedback
Enum error priority: urgent where only high/medium/low allowed Put allowed values close to generation point
Reference error Task ID, source ID, or dependency points to nonexistent object Post-generation reference validator
Semantic error YAML parses but encodes wrong business meaning Domain examples plus human/LLM judge or deterministic rule
Unsafe default Agent invents permissions, regions, owners, or approvals Explicit default policy and reject rules
Overgeneration Adds prose, comments, or extra sections Strict output contract and parser-based rejection
Under-specification Leaves fields blank or generic Ask-for-clarification gate or required-field checklist

Bedrock Prompt Optimization Template

Bedrock Advanced Prompt Optimization expects one JSON object per line. For a YAML-generation task, the prompt template should include the schema contract and the variables should carry the domain request and relevant source context.

{"version":"bedrock-2026-05-14","templateId":"statebench-yaml-task-v1","promptTemplate":"Generate valid YAML for this StateBench task. Use only the schema below. Request: {{request}} Schema: {{schema}} Source context: {{context}}","steeringCriteria":["PRECISE"],"evaluationSamples":[{"inputVariables":[{"request":"Create a source-acquisition task for an AWS Bedrock invocation logging gap."},{"schema":"<schema omitted>"},{"context":"<redacted source context>"}],"referenceResponse":"<validated YAML reference>"}]}

Use Bedrock prompt optimization when the task can be represented as a reusable prompt template, 1-100 high-quality evaluation samples cover the important variants, and a reference response, custom judge, or Lambda evaluator can measure output. Do not use it as a replacement for a larger fine-tuning corpus. It is an optimization harness.

DSPy/GEPA Pattern

GEPA-style optimization is useful when the system can emit rich textual feedback. YAML validators are unusually good feedback generators.

Minimal loop:

  1. Define a DSPy signature for request, schema, context -> yaml.
  2. Run examples through the generator.
  3. Parse YAML.
  4. Validate schema.
  5. Validate domain references.
  6. Convert each failure into feedback text.
  7. Optimize the instruction/program against a metric such as exact structural validity plus semantic task score.
  8. Freeze the optimized prompt and rerun on held-out examples.

Metric sketch:

score =
  0.35 * parses_as_yaml
+ 0.35 * passes_schema
+ 0.20 * passes_domain_rules
+ 0.10 * concise_no_extra_text

The important part is that GEPA sees why an output failed, not just that it failed.

lm-evaluation-harness Pattern

lm-evaluation-harness is useful for repeatable model comparisons, especially when local or hosted models should be evaluated under the same task contract. The task YAML should make prompt construction, dataset loading, generation settings, few-shot count, and scoring explicit enough to reproduce.

For structured YAML generation, the harness layer should log samples and call a postprocessor that parses and validates the generated YAML. The reported metric should be validator-driven rather than string-match-only.

Skeleton task concept:

task: domain_yaml_generation_v0
dataset_path: json
dataset_kwargs:
  data_files:
    validation: domain_yaml_eval.jsonl
output_type: generate_until
training_split: null
validation_split: validation
doc_to_text: !function prompts.doc_to_text
doc_to_target: "{{reference_yaml}}"
generation_kwargs:
  until:
    - "\n\n"
metric_list:
  - metric: yaml_schema_pass
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1
  num_fewshot: 0

This style does not replace Bedrock or DSPy optimization. It creates a stable scoreboard for the baseline, optimized prompt, and any later fine-tuned model.

Trace Mining Pipeline

Mine local and production traces for YAML generation examples only after redaction and labeling:

  1. Identify episodes where the final accepted artifact is YAML.
  2. Capture the request, schema version, relevant retrieved context, generated YAML, validator output, edits/retries, and final accepted YAML.
  3. Drop examples with ambiguous acceptance or missing validation.
  4. Deduplicate near-identical records by schema branch and request intent.
  5. Split by scenario, not random line count, so variants do not leak from train to eval.
  6. Keep hard failures as eval cases, not just training cases.
  7. Store provenance: source session, date, schema version, validator version, redaction version, and human label.

When To Fine-Tune

Fine-tune only when at least one of these is true:

  • The target model repeatedly violates the same structure despite constrained decoding and validator feedback.
  • The domain has compact, repeated transformations that are expensive to describe in every prompt.
  • Latency/cost requires a smaller model to perform without long instructions.
  • House style or domain shorthand needs to be internalized across many tasks.
  • The same schema family will be generated often enough to amortize curation.

Do not fine-tune when the problem is missing retrieval context, an underspecified schema, weak validation, or bad examples. Fine-tuning amplifies those defects.

For each YAML family, maintain:

  • schema.yaml or schema.json;
  • validator.py or equivalent deterministic checker;
  • eval.jsonl with held-out requests and reference YAML;
  • train_candidates.jsonl with trace-mined examples not yet approved;
  • train_approved.jsonl with reviewed examples;
  • bedrock_prompt_optimization.jsonl;
  • lm_eval_task.yaml;
  • results/ with baseline, optimized, and model comparison runs;
  • README.md explaining schema version, acceptance policy, and known failure modes.

Sources

Brandon Sneider | brandon@brandonsneider.com June 2026