Finance/Quant AI Objective Coverage Audit

See also: finance-quant-ai-coverage-audit-2026.md · finance-quant-source-ingestion-queue-2026.md · statebench-model-candidate-registry-2026.md · finance-model-runtime-bakeoff-v0 · finance-source-discovery-eval-v0 · finance-sentiment-routing-v0

Document role: internal completion audit for the active finance/quant AI research goal. It maps the requested end state to current repo evidence, distinguishes proven coverage from remaining work, and keeps the scope from collapsing into only source collection or only benchmark plumbing.

Objective Requirements

The active objective decomposes into eight concrete requirements:

1. identify what is missing in finance and quant AI across industry, papers, whitepapers, reports, podcasts, articles, and official social/talent signals;
2. determine what is working and what is not working at systematic quant shops;
3. capture leakage from podcasts, articles, official pages, job listings, and LinkedIn/company social without overclaiming alpha;
4. ingest valuable sources through the repo’s existing PDF, podcast, source-card, wiki, and StateBench pipelines;
5. separate live web/source discovery from deterministic benchmark scoring;
6. identify interesting domain-specific models and SLMs not previously tracked;
7. record backend/runtime limits across MLX, llama.cpp/GGUF, CUDA, AMD/ROCm, API, SageMaker, Bedrock, and AWS Neuron/Inferentia/Trainium;
8. prove the work with executable checks, scorecards, or explicit evidence tables rather than loose notes.

Coverage Matrix

Evidence Strength

What We Are Still Missing

The remaining gap is measured execution, not source awareness. The corpus now has enough finance/quant industry, model, benchmark, podcast, report, and official-practitioner evidence to start answering the strategic question. What is still missing is proof that specific models and backends can do the work better, cheaper, or more reliably.

The highest-value unfinished items are:

1. First scored model finance-replay-v0 run. Dry-run rows now prove manifest and scorecard aggregation, and the full oracle expected-patch control now proves 118/118 suite/check executability across source acquisition, wiki maintenance, and reflect-audit tasks. The current suite adds ra-123 for Balyasny multi-strat factor-agent caveats and sa-083 for QFBench executable quant-code benchmark evidence. The refreshed frontier/API cost estimate is $7.512744; next run a frontier API control after explicit cost_cap, or one current open/local candidate after endpoint smoke. A fresh qwen3-6-35b-a3b-hf localhost smoke artifact shows localhost:8000 is not currently serving the model, so the Qwen route is still blocked on backend_memory_check. Summarize with the scorecard runner as non-oracle model/backend rows.
2. First approved model preflight. Fill one preflight record for a candidate that can actually run in the current environment; do not approve Neuron until there is real Inferentia/Trainium compile evidence. The current readiness note ranks the first viable paths: frontier API is blocked only on cost_cap; cpa-qwen3-8b-v0 has all source checks passed but needs an approved endpoint; Kuvera now has a preflight-gated advisor/portfolio endpoint runner path but still needs runtime smoke; Qwen3.6/Gemma need endpoint-memory smoke; regional SLMs now have a preflight-gated endpoint runner path but still need endpoint smoke and preflight approval.
3. First quant-code model/backend row. The new finance-quant-code-v0 seed fixture records the QFBench/QuantEval execution discipline, scores a static answer artifact 8/8, and runs a tiny deterministic execution smoke 3/3. The missing layer is a real model/backend row, then a larger code-execution harness with realistic frozen inputs, task-local data, verifier, oracle/reference solution, repeated runs, one-shot baseline, tool traces, cost, latency, and failure-mode capture.
4. First retrieval/SQL backend score. The static answer-artifact control now records 46/46 in statebench/results/finance-retrieval-sql-v0/sample-pass-artifact.summary.json. The missing layer is a preflight-approved model/backend row, followed by Postgres and Snowflake proof. Local Postgres preflight is captured and currently fails only on missing driver availability (psycopg, psycopg2, or pg8000), so the next environment step is installing a driver and pointing the harness at a disposable database. The Snowflake semantic contract now has a passing local fixture-alignment artifact, but the missing Snowflake layer is still real Cortex/warehouse execution with RBAC, traces, and cost metadata.
5. Sentiment/routing next candidate. The YorkFr, ModernFinBERT, and ProsusAI FinBERT local MPS rows exist. ModernFinBERT leads label-only acceptance at 11/15, while YorkFr and ProsusAI accept 10/15; all score 0/15 accepted on full workflow routing. The qwen3-grpo-parser-contract row scores 15/15 as a parser/output-contract smoke, not as a model-quality result. Next compare FinSenti after the Qwen3.5/FinSenti adapter or GGUF backend is proven.
6. Official social hardening. Convert official LinkedIn/company-social discoveries into source cards only when the account, URL, date, and archive/screenshot or primary-source backing are preserved. The finance-talent-leakage-v0 static-control scorecard now records 10/10 for the promotion-policy fixture, and sa-073 makes this a first-class source-discovery family rather than a buried subcase of firm/careers pages. The remaining work is scoring real browser/source-discovery agents and adding new frozen tasks only when official evidence packs are captured. 6a. Vendor product-signal scoring. The new finance-vendor-product-signal-v0 fixture covers QuantConnect-style research pipelines, assistant teams, incumbent terminal AI, governed connector ecosystems, AI-native workflow platforms, named-customer quote caveats, podcast source-discovery routing, terminal-replacement overclaims, vendor metrics, MCP/tool surfaces, and wiki promotion policy. sa-082 now makes this a materialized source-acquisition fixture family in finance-source-discovery-eval-v0. The remaining work is scoring real models on it and only expanding it when a new vendor pattern appears. The runner now has a preflight-gated endpoint path, and the dedicated frontier-api-vendor-product-signal-control preflight exists, but this lane still needs cost estimate/cap and endpoint/API-route evidence before a non-static model/backend row can be produced. 6b. Agentic factor-discovery scoring. The new finance-agentic-factor-discovery-v0 fixture covers Hubble/FactorEngine style source-to-factor conversion, safe DSL/AST execution, point-in-time validation, diagnostics, turnover/cost/capacity controls, diversity/crowding, neutralization, walk-forward promote/hold/retire decisions, narrative-bias rejection, and runtime portability. The remaining work is a preflight-approved model row plus a deterministic toy factor-code harness with point-in-time data, reference results, cost/capacity checks, and failure traces. 6c. Portfolio/risk-construction scoring. The new finance-portfolio-risk-construction-v0 fixture covers screened-universe audits, covariance/risk experiment design, graph-construction critique, constrained weights, action-first RL policy, risk-aware multi-agent controls, inverse-RL evidence limits, backtest skepticism, and runtime portability. The remaining work is a preflight-approved model row plus deterministic covariance/weighting and graph-construction harnesses with point-in-time data, reference outputs, turnover/cost/capacity checks, and survivorship-bias controls.
7. Negative-evidence task expansion. The first expansion is now materialized in ra-119 through ra-123: Acadian process overclaim, Two Sigma temporal leakage, Winton selection bias, cross-firm autonomous agent-alpha collapse, and Balyasny factor-agent workflow caveats. sa-085 adds the Senate/HSGAC regulatory source-acquisition counterpart for system-definition consistency, human-review boundary, test cadence, version control, disclosure adequacy, and audit-trail preservation. The remaining work is scoring real candidates and adding more tasks only when new practitioner or regulatory evidence introduces a distinct failure mode.
8. Domain SLM bakeoff. Score at least one current Qwen-family artifact, one Gemma-family artifact, one Liquid task specialist, one finance reasoning specialist, one accounting/compliance specialist on finance-accounting-compliance-v0, one advisor/personal-finance specialist on finance-advisor-portfolio-v0, one treasury/payments/liquidity row on finance-treasury-payments-liquidity-v0, one agentic factor-discovery row on finance-agentic-factor-discovery-v0, one portfolio/risk-construction row on finance-portfolio-risk-construction-v0, one multimodal finance VLM, one browser/source-discovery agent such as Fara-7B or Fara1.5, one regional/language finance SLM where a matching fixture exists, one local GGUF finance-reasoning artifact such as Qwen3-8B finance GGUF, one retrieval/reranking specialist or tiny embedding artifact on the appropriate suite, and one finance-native time-series market model such as Kronos only after finance-time-series-market-v0 has locked data, leakage controls, baselines, transaction-cost/capacity assumptions, and runtime smoke proof.
9. Podcast and article quality triage. Keep adding only named-practitioner, source-backed media to findings; vendor/product podcasts and social posts stay as discovery unless they expose architecture, process controls, failure modes, or named customer evidence.
10. Kaggle and competition harness extraction. Preserve competition metrics, split rules, leakage controls, kernels/notebooks, and local graders as eval design evidence; do not treat competition rankings as finance-alpha proof. The finance-kaggle-artifact-v0 static-control scorecard now records 10/10 for official metadata, metric/split, leakage, baseline, notebook, MLE-harness, community-timing, git-boundary, and not-alpha checks. The remaining work is scoring real browser/source-discovery agents and building larger executable competition-style harnesses only when frozen task-local data, reference solutions, leakage controls, runtime traces, and git exclusions are present.
11. Treasury/payments execution harness. The finance-treasury-payments-liquidity-v0 static-control suite now covers the regulated money-movement policy boundary. The remaining work is a preflight-approved model row plus synthetic mandate/invoice/statement/ledger fixtures and deterministic tool traces for authorization, sanctions/fraud, settlement, reconciliation, kill-switch, and audit controls.
12. Investment-banking/deal-work execution harness. BankerToolBench is now frozen as source-acquisition fixture sa-086, but the repo still needs a deterministic StateBench task family for senior-banker request parsing, data-room provenance, market/SEC research, Excel formula integrity, deck/memo generation, cross-artifact reconciliation, confidentiality/MNPI, and human-review handoff.

Current Interpretation

What top finance and quant shops appear to be doing publicly:

• using AI to accelerate research, coding, document analysis, feature discovery, data exploration, and analyst/PM workflows;
• building governed internal platforms around data lineage, entitlement checks, notebooks, backtests, logs, and evaluation gates;
• using ML at industrial scale for market prediction, execution, risk, storage, and low-latency deployment;
• treating broad chat assistants as shallow unless embedded in firm-specific tools, permissions, and workflows.

What is not proven or not working reliably:

• public sources do not independently prove genAI-driven live alpha;
• general agents can widen the hypothesis funnel and worsen false discovery;
• LLMs introduce historical knowledge-cutoff and temporal-leakage risks;
• finance artifacts can look plausible while formulas, state, citations, or execution traces are wrong;
• vendor product pages are architecture/product signals unless named customers or reproducible measurements are present.

Completion Standard

The objective should not be marked complete until current repo evidence proves:

1. promoted source batches have raw source records or explicit reject/watchlist decisions;
2. source-discovery categories have frozen StateBench fixture coverage;
3. domain-model candidates have runnable manifest entries and preflight gates;
4. at least one scored model/runtime run exists for finance-replay-v0;
5. at least one scored retrieval/SQL answer-artifact run exists, and at least one model/backend retrieval/SQL row exists with backend metadata and preflight validation;
6. docs and wiki pages distinguish workflow evidence from alpha/performance evidence;
7. validation commands for the touched runners, manifests, and fixtures pass.

Until those scorecards exist, the correct status is: research map and ingestion control plane are strong; model/runtime recommendations remain unmeasured.

Next Verification Commands

Use these checks before claiming another layer is complete:

python3 -m statebench.runners.source_discovery_eval \
  --queue statebench/suites/finance-source-discovery-eval-v0.json \
  --validate-only

python3 -m statebench.runners.source_discovery_eval \
  --queue statebench/suites/finance-source-discovery-eval-v0.json \
  --section fixture-coverage \
  --format markdown

python3 -m statebench.runners.finance_retrieval_sql_fixture \
  --fixture-dir statebench/fixtures/finance-retrieval-sql/v0 \
  --answers-file statebench/fixtures/finance-retrieval-sql/v0/sample_answers/pass.json \
  --format markdown

python3 -m statebench.runners.finance_retrieval_sql_fixture \
  --fixture-dir statebench/fixtures/finance-retrieval-sql/v0 \
  --snowflake-contract-check \
  --format markdown

python3 -m statebench.runners.model_candidate_manifest \
  --manifest statebench/suites/finance-model-runtime-candidates-v0.json \
  --format markdown

python3 -m statebench.runners.suite_runner \
  --suite-id finance-replay-v0-oracle-full-20260601-expanded118 \
  --suite-file statebench/suites/finance-replay-v0.json \
  --backend frontier-api \
  --model oracle-expected-patch \
  --policy overwrite \
  --oracle-expected-patch