StateBench Finance Replay Model Candidate Registry

Source ledger: sources/21-benchmarks/statebench-model-candidate-registry-2026-raw.md

This registry turns the finance/quant model landscape into executable StateBench lanes. It is not a leaderboard. A candidate enters the registry when there is a verified artifact, model card, primary paper, vendor engineering post, or already-ingested source that tells us exactly what to run and what cannot be assumed.

Machine-readable candidate manifest: statebench/suites/finance-model-runtime-candidates-v0.json.

Execution queue: statebench/suites/finance-model-runtime-bakeoff-v0.md and statebench/suites/finance-model-runtime-bakeoff-v0.json.

Operating Rule

Evaluate artifacts, not brand names. A base HF checkpoint, GGUF quant, MLX conversion, API route, SageMaker endpoint, and Neuron-compiled artifact are separate candidates. A model that works under CUDA or vLLM is not automatically compatible with MLX, llama.cpp, ROCm, or AWS Neuron.

Qwen2.5-era artifacts are allowed only as specialist controls, especially for Snowflake/Text2SQL where the current runnable specialist lineage still matters. They are not defaults for general StateBench reasoning.

Ollama is excluded. llama.cpp/GGUF remains in scope.

Candidate Lanes

Current Preflight Update

The dedicated frontier/API control for finance-vendor-product-signal-v0 now has a small-suite cost estimate at statebench/results/finance-vendor-product-signal-v0/frontier-api-cost-estimate-20260605.json. The exact 12-task backend prompt is 904 input tokens including the system instruction, with a 4,096 output-token cap. A conservative high-cap pricing profile at $15/M input and $75/M output estimates $0.320760 before retries and records a recommended $1.00 cap.

This is preflight evidence only. The control is still not approved for a scored run until an explicit cost_cap and a concrete endpoint/API route are recorded. It also remains API-only evidence and does not imply MLX, GGUF, CUDA, ROCm, SageMaker, Bedrock, or Neuron portability.

sprocket-gex-deepseek-r1-qwen14b-lora is now a watchlist-only options/GEX adapter candidate. Its preflight record pins the adapter revision and artifact files, but scored use is blocked until base-license review, adapter-load smoke, a structured GEX schema fixture, and output-hygiene checks are recorded. Treat it as a derivatives/options schema classifier, not a finance-reasoning, trading-action, alpha, portfolio, or risk-management model.

First Real Run Order

Use the machine-readable candidate manifest before starting a run so artifact format, backend, required preflight, and negative compatibility constraints are recorded consistently.

Command generation:

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --emit-suite-commands finance-replay-v0

Those emitted commands are dry runs by default and should parse without model access. Add --scored only after exact revision, hardware, context length, and backend constraints are replaced with verified values.

For batch review, add --emit-batch-script. Dry-run batch commands still write StateBench result directories if executed, so use generated commands with --list-tasks when only checking suite selection.

Target-suite validation:

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --suite-target-report

Readiness report:

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --readiness-report

Artifact preflight template:

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --preflight-template \
  --artifact-id qwen3-6-35b-a3b-hf

Write and validate preflight records:

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --preflight-template \
  --write-preflight-template \
  --artifact-id qwen3-6-35b-a3b-hf

scripts/.venv-tts/bin/python3 -m statebench.runners.model_candidate_manifest \
  --validate-preflight-record statebench/preflights/qwen3-6-35b-a3b-hf-preflight.json

The validator enforces statebench/schemas/preflight_record.schema.json before checking whether all required preflight items have evidence. Scored finance-replay-v0 suite runs and finance-retrieval-sql-harness-v0 backend calls require --preflight-record; dry-runs, --list-tasks, fixture smoke checks, prompt export, Postgres preflight, and scoring an existing answer artifact remain available without approval.

1. Run a frontier API control against all 118 finance-replay-v0 tasks.
2. Run Qwen/Qwen3.6-35B-A3B through an OpenAI-compatible vLLM or SGLang endpoint, with exact HF revision, 262K effective context unless extended context is deliberately enabled, and a thinking-mode note.
3. Run google/gemma-4-31B or the best Gemma 4 artifact that fits available hardware; record whether the run is dense, MoE, API, GGUF, or MLX.
4. Run one Liquid task-specialist artifact on the subset it should plausibly improve: source-pack QA, transcript cleanup, extract/PII, RAG, tool routing, or reranking. Do not force it into general finance reasoning.
5. Run one finance specialist only after its license and artifact are verified. rLLM-FinQA-4B, ODA-Fin-style Qwen3 finance post-training, and TheFinAI/Fin-o1-8B are higher signal than old BloombergGPT/PIXIU references because they are closer to local StateBench experiments. khazarai/Fino1-4B is now the cheaper Qwen3-4B FinQA-style control in the same specialist lane, but it remains blocked on license-conflict and runtime preflight checks. 5a. Do not leave new static-control suites without candidate artifacts. The manifest now targets finance-treasury-payments-liquidity-v0, finance-agentic-factor-discovery-v0, and finance-portfolio-risk-construction-v0 with frontier/API ceiling control, Qwen3.6 open-base comparison, Gemma 4 where the task is factor/portfolio reasoning, and Liquid Financial Analyst only as a treasury/payments tiny-model cost-floor control. These are planning rows, not model scores. 5b. Treat Trading-R1 / Alpha-R1 / Trade-R1 as a separate paper-first trading/factor RL lane. They are highly relevant to systematic quant research because they target structured trading theses, factor screening, noisy market rewards, and reward hacking, but they should not enter a scored runtime queue until code, weights, license, exact revision, backend, transaction-cost assumptions, point-in-time data, and reproducibility evidence are recorded.
6. Use finance-sentiment-routing-v0 before using large chat models for headline or earnings-snippet classification. The 15-task held-out fixture and static gold-control scorecard now exist. FinSenti-Qwen3.5-9B is the current Qwen-family candidate to compare against FinBERT/ModernFinBERT and tiny Qwen3-0.6B sentiment controls such as YorkFr/financial-sentiment-qwen3-v2, but it is not runnable in this repo until the Qwen3.5 image-text-to-text adapter, FinSenti tag parser, or llama.cpp/GGUF backend is proven. 6a. Add an earnings-call evasion sublane. Eva-4B-V2 is a Qwen3-4B-derived specialist for analyst-question / management-answer directness and evasiveness. It belongs in transcript/disclosure-quality triage and earnings-call feature extraction, not in generic sentiment, alpha, investment advice, or portfolio construction. Any score must record label-source bias, LLM-as-judge risk, intermediate-class subjectivity, data-vintage drift, and human-review requirements.
7. Build the accounting/compliance guard lane separately from broad finance reasoning. CPA-Qwen3 belongs on accounting/tax/audit boundary tasks, while FinGuard belongs on policy-grounded verifier tasks after jurisdiction and policy corpus are defined.
8. Build the multimodal finance lane separately from the text patch runner. AITRADER/Amsi-fin-o1.5-mxfp8-MLX is useful for chart/OCR/screenshot source tasks only after an mlx-vlm smoke test proves it runs locally.
9. Build the retrieval lane separately from the patch runner. Start with Qwen3 Embedding plus Qwen3 Reranker against FinRetrieval-style and internal source-localization tasks, then compare finance-specific/gated encoders such as FinE5 if licensing permits.
10. Build the SQL lane separately from the patch runner. Treat Snowflake Arctic-Text2SQL-R2 as the current Snowflake-specialist recipe to test, but do not score vendor-post claims as internal performance.
11. Only after base/API/local runs identify repeatable failure modes, compare Unsloth, LLaMA-Factory, AWS native, and MLX fine-tuning paths.

Required Manifest Metadata

Every run must fill these fields in statebench/results/<suite_run_id>/manifest.json:

• model
• weights_revision
• artifact_id
• adapter_id
• merge_state
• artifact_format
• artifact_precision
• backend_id
• backend_hardware
• backend_constraints
• context_length

For backend constraints, record negative knowledge explicitly, for example:

• neuron compatibility unproven until operator and compilation preflight passes
• mlx conversion not tested
• gguf quantization is community artifact, not upstream release
• api-only proprietary route
• license review required before commercial use

Why This Matters For The Finance Goal

The repo already has broad finance/quant coverage. The remaining risk is selection drift: choosing models because they are fashionable, locally convenient, or vendor-promoted instead of because they pass the finance tasks we actually need.

This registry prevents three common mistakes:

1. Treating Qwen, Gemma, or Liquid family names as scores.
2. Letting SQL/retrieval/source-discovery performance leak into the general patch-runner benchmark.
3. Assuming deployment portability across CUDA, MLX, llama.cpp, AMD, and Neuron without explicit artifact evidence.

Current Interpretation

As of May 29, 2026:

• Qwen/Qwen3.6-35B-A3B is the default current open Qwen-family base candidate for StateBench finance replay. It is current enough to avoid the stale Qwen2.5 trap and has explicit long-context and agentic-coding evidence.
• Gemma 4 is the right open-family counterweight to Qwen, especially for multimodal and long-context document tasks.
• LiquidAI is most interesting as a family of small task specialists and portable runtime artifacts, not as a finance alpha/reasoning model.
• Arctic-Text2SQL-R2 is the most important new SQL signal because it argues that compact, domain/dialect-trained models can beat frontier systems on enterprise SQL. That belongs in a Snowflake/Postgres harness, not the general finance replay patch runner.
• Dev9124/qwen3-finance-model is a current Qwen3 finance-instruction watchlist artifact with Apache-2.0 model-card licensing and vLLM/SGLang runtime snippets. It is useful as a broad fine-tune control, but model-card evidence alone is weaker than ODA-Fin/rLLM-style paper-backed specialists.
• TheFinAI/Fin-o1-8B is now tracked as a Qwen3-8B finance-reasoning candidate. The current HF card supersedes the stale local Llama-lineage note; score it as a concrete Qwen3 artifact only after exact revision and runtime proof.
• khazarai/Fino1-4B is now tracked as a separate Qwen3-4B finance-reasoning watchlist candidate, not a smaller alias for TheFinAI/Fin-o1-8B. The captured HF revision is 15c0d969d0bd2aa6515e0e315b5aecc063c2d7ee, architecture is Qwen3ForCausalLM, base metadata is unsloth/Qwen3-4B, and the model has 4,022,468,096 BF16 safetensors parameters. Its dataset points to TheFinAI/Fino1_Reasoning_Path_FinQA at revision 0316559663c46e50d90e75f6e14a6d8f679cdba8 under cc-by-4.0. Block production/commercial assumptions until the model license conflict is resolved: HF metadata says Apache-2.0 while the README body says MIT. Treat it as a cheap FinQA-style numerical/table-QA control only; no GGUF, MLX, ROCm, SageMaker, Bedrock, or Neuron portability is proven.
• CPA-Qwen3 and FinGuard widen the domain-specialist lane from investment reasoning into accounting/compliance and policy-grounded guardrails. Their scores should live on accounting/compliance fixtures, not generic replay. The first finance-accounting-compliance-v0 seed fixture now exists with 10 guardrail tasks and a static 10/10 answer-artifact control; model/backend scores still require preflight approval.
• AITRADER/Amsi-fin-o1.5-mxfp8-MLX adds a current MLX finance VLM candidate for charts, OCR, screenshots, and multimodal source triage. It is not proof that MLX should replace other runtimes, and it is not a text-only benchmark default.
• YorkFr/financial-sentiment-qwen3-v2 is a tiny MIT-licensed Qwen3-0.6B sentiment artifact. It belongs in a held-out sentiment/routing lane, not the general finance replay patch runner. The fixture and Transformers-local adapter now exist. The first local run is negative evidence: 0/15 accepted, despite partial label signal, because route/rationale controls failed.
• Regional and narrow finance SLMs are now tracked as a separate watchlist: iravikr/qwen3-0.6b-finance-india for India budget/policy QA, KBTG-Labs/THaLLE-0.2-ThaiLLM-8B-fa for Thai/English finance-language coverage, and Ayansk11/qwen3-4b-financial-sentiment-grpo as a smaller Qwen3 sentiment/GGUF candidate. sweatSmile/Qwen3-4B-Instruct-FinanceQA is a useful cautionary control because its Hugging Face metadata reports a Qwen2.5 base despite the repository name.
• The May 31 HF watchlist pass adds a more concrete local runtime candidate: mradermacher/Qwen3-8B-finance-V1.0.1-Merged-V2-GGUF, revision 29d21ffe58d0e9c7f86136fe7f9eaab3eb0d1332, as a llama.cpp/GGUF finance reasoning control. It should be scored as its exact quant file, not as the upstream RinKana merge. The same pass adds PeterAM4/Qwen3-Embedding-0.6B-GGUF, revision bb661aeeeafa4ff7303b5f5e1e80255efd6a513c, as a tiny local embedding candidate with finance-related imatrix/data tags.
• The June 5 HF watchlist follow-up adds shm2290/finance-embeddings-gemma-300m-v2 as a finance-specific EmbeddingGemma 300M control. It is useful precisely because it is small and retrieval-only: test it on source localization, wiki/static-index retrieval, SEC/investor-document paragraph retrieval, and podcast/source-card metadata search. Do not use it as a finance reasoner, source-discovery agent, accounting model, or alpha signal. Its license inherits from google/embeddinggemma-300m, so commercial and redistribution use need base license review before promotion.
• The June 6 HF watchlist follow-up adds souflex56/qanchor-reranker-qwen3-0.6b-merged as a Chinese financial-document reranker candidate. It is useful because it targets A-share annual reports and related filings with a current Qwen3 sequence-classification reranker, but it is not a generator or broad finance model. Score it only after a pinned Chinese financial-document retrieval fixture, first-stage BM25/embedding/RRF baseline, preserved qwen3_template formatting, same-document scope check, and reranker trace schema are in place.
• The June 6 regional-sentiment follow-up adds p988744/eland-sentiment-zh-vllm as a Qwen3-4B Traditional Chinese/Taiwan financial-sentiment routing candidate. It is useful for Taiwan stock-market headlines, entity sentiment, and opinion sentiment, but it is not a U.S. SEC, accounting, quant-research, investment-advice, portfolio, trading, or alpha model. Score only after exact HF revisions are captured, the required system prompt is preserved, a held-out Traditional Chinese/Taiwan fixture exists, and vLLM/PEFT/GGUF artifacts are separated. Do not use Ollama.
• The June 6 structured-extraction follow-up adds lmxxf/financial-report-lora-qwen3-14b as a Chinese financial research-report metric-extraction adapter on Qwen/Qwen3-14B. It is useful because it targets schema-checked extraction of deeply analyzed metrics from Chinese report paragraphs, a better fit for source cards and factor-discovery bootstrapping than generic sentiment. It is blocked on license visibility, base revision, PEFT adapter-load smoke, held-out Chinese report metric spans, JSON-schema validation, and output-hygiene checks. Do not treat it as a broad finance reasoner, accounting authority, alpha model, or runtime portability proof.
• Mikkkkoooo/qwen35-4b-private-analyst-full-corpus and DMindAI/DMind-3-mini are recorded as watchlist controls only. The former is a private-corpus analyst-style fine-tune with license other; the latter is a Web3/DeFi finance model, not a U.S. asset-management or SEC-filing model.
• The May 31 follow-up pass adds three narrow candidates. Mihenk is a Qwen3.6-35B-A3B Turkish/BIST finance model now covered by the finance-regional-language-v0 scope-control fixture. Distil/LFM2.5 voice assistant is a 354M banking tool-calling candidate for voice/tool-routing fixtures, not quant research. TokenFactory Gemma SEC extraction v3 is a GGUF/llama.cpp structured-extraction candidate with unverified card metrics and Gemma-license constraints.
• The June 5 follow-up records a separate community GGUF conversion of Mihenk. This improves the local llama.cpp runtime lane but does not change the upstream HF/BF16 evidence. The GGUF artifact is blocked until an exact file, prompt-template compatibility, and llama.cpp smoke test are recorded.
• The June 5 Gemma follow-up records Sengil/turkish-gemma-9b-finance-sft as a regional Turkish finance-language Gemma2 adapter. It is useful as a family-control against the Qwen/Mihenk Turkish lane, but it remains blocked on base-license review, adapter-load smoke, and visible reasoning/output-hygiene checks.
• The second May 31 follow-up pass adds three more executable local controls. Myyyyyyyyyyyyyy/qwen3-14b-bookkeeper-gguf fills a bookkeeping/accounting workflow gap with Qwen3-14B GGUF and explicit work-in-progress caveats. mad-lab-ai/qwen3-1.7b-sentiment-gguf is a tiny local sentiment-cost-floor candidate for label/routing fixtures, with leakage and alpha-overclaim guardrails. vividdream/Qwen-Open-Finance-R-8B-IQ4_NL-GGUF gives a multilingual finance/economics llama.cpp control that must be scored separately from upstream DragonLLM weights and other quantizations.
• neoyipeng/ModernFinBERT-base and ProsusAI/finbert establish label-only encoder baselines on the same fixture. ModernFinBERT currently leads the label slice (0.733 vs. 0.667 for ProsusAI and YorkFr), but both encoders fail full routing acceptance because they do not emit route/rationale policy.
• AssetOpsBench QLoRA tool-knowledge evidence strengthens the fine-tuning gate: small models may internalize stable tool catalogs, but only after StateBench shows repeated tool/planning failures and after catastrophic forgetting is measured.
• Qwen3.7-Max is worth tracking as a proprietary long-horizon agent reference, but it is not an open/local candidate and should not displace Qwen3.6 for the local artifact lane without verified open weights.
• Relational Probing adds a distinct market-graph lane: Qwen3 0.6B/1.7B/4B SLMs are used as upstream hidden-state encoders whose relation head induces financial-entity graphs for downstream stock-trend prediction. This is closer to quant modeling than finance chat, but it is paper-first evidence until code/data/checkpoints are verified and leakage-safe point-in-time fixtures exist.