Finance Measured Runtime Evidence Ledger

Brandon Sneider · June 2026

> **Document role:** separate measured model/runtime evidence from static

Document role: separate measured model/runtime evidence from static controls, parser-contract checks, dry runs, source cards, and preflight readiness. This prevents the finance/quant model watchlist from being read as performance evidence.

Current Measurement State

The repo now has strong source coverage, many static-control fixtures, and a large candidate manifest. It still has very little real model/runtime evidence.

Evidence Type	Current Evidence	What It Proves	What It Does Not Prove
Real local model rows	`finance-sentiment-routing-v0` ModernFinBERT, ProsusAI FinBERT, and YorkFr Qwen3 sentiment summaries	Local Transformers/MPS scoring path works for narrow sentiment labels	Full workflow routing, alpha, investment signal quality, source acquisition, SQL, accounting, or portfolio competence
Real local MLX tool-routing row	`finance-tool-routing-v0/functiongemma-270m-it-4bit-mlx.answers.summary.json`	Cached MLX model loading and scored local Metal path work through `mlx_lm`; malformed-output failures are captured as scoreable artifacts	Tool-routing competence, finance reasoning, retrieval, SQL, accounting, portfolio, source-discovery, advice, or alpha
Parser-contract row	`qwen3-grpo-parser-pass.summary.json`	The answer contract and scorer can accept perfect structured routing output	That a deployed Qwen model generated the answers
Static gold controls	Sample-pass rows across accounting, advisor, regional, time-series, quant-code, talent, Kaggle, vendor, treasury, factor, and portfolio suites	Fixture/rubric/scorecard plumbing is internally coherent	Model quality, backend quality, runtime portability, production readiness, or finance alpha
Local execution harness sanity	`finance-quant-code-v0/sample-execution-artifact.summary.json` with `local-python-subprocess`	Tiny deterministic code-execution checks can run and aggregate locally	QFBench performance, model-generated code quality, trading readiness, or production data-rights compliance
Runtime smoke / preflight blockers	`openai-runtime-smoke-qwen36-localhost8000-20260531.json`, preflight JSON records, bakeoff phase report	The repo records why model/backend scoring is blocked	That any blocked model is good or bad
Local runtime probes	`amsi-fin-o1-5-mxfp8-mlx-preflight.json` and `qwen-open-finance-r-8b-iq4-nl-gguf-preflight.json` updated on 2026-06-02	MLX core/GPU initializes in the repo venv; the selected `mlx_lm.generate` CLI still crashes before model load; GGUF is blocked locally by missing llama.cpp runtime	That MLX or GGUF model quality has been measured

Real Local Model Rows

The only current real local model comparisons are sentiment/routing rows.

Model	Backend	Mode	Accepted	Label Accuracy	Route Accuracy	Abstention Accuracy	Interpretation
`neoyipeng/ModernFinBERT-base`	`transformers-sequence-classifier-local-mps`	label-only	11/15	0.733	0.000	0.800	Best current local label-only classifier. Useful as cheap sentiment input, not as router.
`ProsusAI/finbert`	`transformers-sequence-classifier-local-mps`	label-only	10/15	0.667	0.000	0.800	Older FinBERT control. Similar abstention behavior, weaker label accuracy than ModernFinBERT.
`YorkFr/financial-sentiment-qwen3-v2`	`transformers-local-mps`	label-only	10/15	0.667	0.200	0.733	Qwen3 sentiment fine-tune is not better than ModernFinBERT on this fixture.
`neoyipeng/ModernFinBERT-base`	`transformers-sequence-classifier-local-mps`	full routing	0/15	0.733	0.000	0.800	Fails full workflow routing because classifiers do not emit route/rationale/required terms.
`ProsusAI/finbert`	`transformers-sequence-classifier-local-mps`	full routing	0/15	0.667	0.000	0.800	Same boundary: label model, not workflow router.
`YorkFr/financial-sentiment-qwen3-v2`	`transformers-local-mps`	full routing	0/15	0.667	0.200	0.733	Useful negative evidence for full routing despite current-family Qwen branding.

Real Local MLX Tool-Routing Row

mlx-community/functiongemma-270m-it-4bit was scored on finance-tool-routing-v0 through the local mlx_lm path on Apple Silicon Metal. The artifact loaded and generated, but did not emit a JSON object for the required answer contract. The runner now records this malformed-output case as a scoreable artifact instead of discarding the run.

Model	Backend	Hardware	Accepted	Failure	Interpretation
`mlx-community/functiongemma-270m-it-4bit`	`mlx_lm_local`	`apple_silicon_metal`	0/10	No JSON object; all tasks missing	Useful negative evidence: a tiny local MLX function model can run, but is not adequate for the current finance ingestion/tool-routing contract.

What Is Working

Static-control suite design is working: multiple finance/quant lanes now have deterministic sample-pass artifacts and scorecards.
Local Transformers/MPS execution works for sequence-classification sentiment models and a small local Qwen sentiment generation row.
Local MLX/Metal execution works when run outside the command sandbox. The new tool-routing runner path can load a cached MLX artifact, capture malformed output, and produce a scorecard row.
The sentiment fixture correctly separates narrow label accuracy from workflow-router acceptance. This is important because finance agents often need routing, abstention, rationale, and no-alpha-overclaim behavior, not just positive/neutral/negative labels.
The quant-code fixture now has a tiny local execution smoke, proving the repo can move beyond answer text into deterministic code execution.

What Is Not Working Yet

No real model/backend row exists for finance-replay-v0.
No real model/backend row exists for retrieval/SQL, accounting/compliance, advisor/portfolio, regional-language, time-series/market, quant-code generation, talent/source discovery, Kaggle discovery, vendor-product signal, treasury/payments/liquidity, agentic factor discovery, or portfolio/risk construction.
No local Qwen3.6 or Gemma endpoint is currently proven. The latest localhost smoke records connection refused.
The first local MLX tool-routing control is negative: FunctionGemma loaded, but refused or failed the JSON contract and scored 0/10.
No MLX VLM smoke pass exists for the Amsi-fin candidate. The current local MLX evidence is mixed: repo venv mlx.core initializes on Device(gpu, 0), but the selected Homebrew mlx_lm.generate route crashes before model load.
No llama.cpp/GGUF finance model row exists yet, even though several GGUF candidates are in the manifest. The current local blocker is concrete: no llama-cli, no llama-server, and no Python llama_cpp module.
No AWS SageMaker, Bedrock, or Neuron row exists. Neuron remains a deployment cost lane only after compile/operator/context/quantization/smoke proof.

Immediate Measurement Implications

Treat ModernFinBERT as the current local label-only baseline for sentiment, not as a workflow router.
Keep YorkFr Qwen3 sentiment as negative evidence against assuming that a current-family finance fine-tune automatically beats older specialist encoders on narrow tasks.
Prioritize one approved endpoint or local runtime smoke over adding more watchlist models. The next material improvement is a scored row, not another model-card row.
For full finance workflow routing, compare a structured-output LLM or a tool-calling specialist such as Nexus TinyFunction only after endpoint/tool schema preflight passes.
For non-sentiment finance tasks, do not claim any model ranking until a real scorecard row exists with artifact revision, backend, hardware, precision, context length, cost/latency, and failure modes.
Treat local runtime probes as preflight evidence only. A successful package import is not a model score, and a failed CLI smoke is an environment blocker unless a model artifact was actually loaded and evaluated.

Next Rows To Produce

Priority	Row	Why
1	Frontier/API `finance-replay-v0` ceiling control after explicit cost cap	First broad measured baseline for source acquisition, wiki maintenance, and overclaim repair
2	CPA-Qwen3 on `finance-accounting-compliance-v0` after endpoint smoke	Closest domain-specific non-sentiment row with source/preflight checks mostly satisfied
3	Nexus TinyFunction on `finance-tool-routing-v0` after tool-schema/runtime proof	Tests whether a small tool-calling model can route finance ingestion work better than label-only classifiers and the failed FunctionGemma MLX control
4	Qwen3.6 or Gemma endpoint smoke, then a small static suite before full replay	Establishes open-base comparison without confusing model-card claims with runtime proof
5	One GGUF llama.cpp finance specialist row	Tests the local cost/latency lane the user explicitly wants, without using Ollama

The GGUF row requires a llama.cpp-compatible CLI/server or Python binding before any model file is downloaded or scored. The MLX VLM row requires a working mlx-vlm or mlx-lm route, exact artifact load, and image+prompt smoke before any chart/OCR source task can be scored.

Guardrails

Do not count static gold rows as model evidence.
Do not count parser-contract rows as model evidence unless the model actually generated the structured output under recorded runtime metadata.
Do not promote label-only sentiment rows to routing, trading, allocation, or alpha evidence.
Do not infer CUDA, MLX, GGUF, ROCm, SageMaker, Bedrock, or Neuron compatibility from any other backend.
Never use Ollama.