← Benchmarks 🕐 9 min read
Benchmarks

Finance Kaggle Competition Artifacts: Missing Medium for Quant Eval Design

Brandon Sneider · May 2026

> **Source credibility: MEDIUM. TIER 2-3.**

See also: finance-domain-models-benchmarks-2026.md · finance-ai-benchmark-cluster-2026.md · …/13-multimodal-sources/finance-quant-audio-video-deep-dive-2026.md

Source credibility: MEDIUM. TIER 2-3. Kaggle competition pages are primary sources for competition metadata, data schema, evaluation rules, and leaderboard structure. Public notebooks and discussions are practitioner signals, not production evidence. Treat all competition results as eval-design evidence, not proof of deployable alpha.

Executive Summary

Kaggle is a missing source medium for the finance/quant corpus. It is not a paper, podcast, vendor case study, or GitHub repo. It is a competition-artifact layer: datasets, metrics, public/private splits, notebooks, discussion threads, and post-competition solution patterns.

Source dates matter here. Finance competitions from prior years are still useful for validation design and leakage vocabulary, but AI-agent claims age quickly. Any claim about agent performance, benchmark state of the art, or model choice should be tied to its publish date and rechecked against current systems before it guides the 2026 benchmark plan.

For this repo, Kaggle is useful for four things:

  1. Eval design: realistic metric selection, private holdout design, and leaderboard overfitting controls.
  2. Leakage vocabulary: time ordering, purged validation, embargoes, public/private splits, target construction, and data revisions.
  3. Practitioner baselines: what strong tabular/time-series competitors actually use when money-like metrics are on the line: gradient boosting, ensembling, careful validation, feature hygiene, and occasionally deep sequence models.
  4. Agent benchmark tasks: find a competition, read the rules, identify the metric, summarize top solution patterns, and produce a reproducible eval harness without claiming live trading value.

A second layer is now more important than the individual competition pages: Kaggle-derived agent harnesses. MLE-bench, MLE-Live/CoMind, and MLE-Dojo turn Kaggle-style work into local or controlled agent evaluations. This matters for the repo because our target is not “win Kaggle”; it is “can an agent read a task, build a valid training/evaluation loop, avoid leakage, produce a submission artifact, and document what happened.”

The key caution: Kaggle finance competitions are usually forecasting games with hidden targets, not institutional trading systems. They teach validation discipline; they do not prove alpha, execution capacity, transaction-cost robustness, or live deployability.

The exact rules and metric formulas should not be scraped casually from the browser shell. Unauthenticated Kaggle pages expose reliable title/description metadata but not always the full evaluation text. For implementation, prefer authenticated Kaggle API exports, archived rules, or local grader code from MLE-bench / MLE-Dojo-style harnesses.

On 2026-05-29, this lane was promoted from source-ledger-only to pipeline-backed evidence for two stable PDFs: Kaggle’s own competition host overview and an EAI JPX/Kaggle LightGBM paper. Both PDFs now live under research/21-benchmarks/raw/ with Chandra OCR and metadata sidecars. The new finding is not that Kaggle proves finance alpha; it is that Kaggle gives us official competition mechanics plus finance-specific baseline evidence that can be turned into StateBench source-discovery and leakage-control tasks.

On 2026-05-31, the lane gained an executable seed fixture: finance-kaggle-artifact-v0. It scores whether a model preserves official competition metadata, metric/split evidence, leakage controls, baseline discipline, notebook status, MLE-style harness requirements, community-artifact timing, large-data/git boundaries, and the “not alpha proof” caveat.

Priority Finance / Quant Competition Set

Competition / Benchmark Sponsor Task Signal Eval Design Value
Jane Street Real-Time Market Data Forecasting Jane Street Predict financial market responders from real-world market data Modern real-time tabular/time-series forecasting; strong StateBench source-discovery task
Jane Street Market Prediction Jane Street Decide whether to trade from anonymized market features Utility-style scoring, action thresholding, and public/private leaderboard drift
Optiver Trading at the Close Optiver Predict US stock closing movements Market microstructure around closing auctions; event-time feature engineering
Optiver Realized Volatility Prediction Optiver Predict realized volatility from order book / trade data RMSPE-style volatility forecasting and order-book feature construction
Two Sigma: Using News to Predict Stock Movements Two Sigma Use news analytics to predict stock price performance Text + market data fusion; early public example of alternative-data forecasting
Two Sigma Financial Modeling Challenge Two Sigma Financial modeling / prediction competition Historical signal for quant competition culture and validation caveats
G-Research Crypto Forecasting G-Research Predict real crypto market data Weighted correlation, cross-asset time-series, and non-equity market structure
JPX Tokyo Stock Exchange Prediction JPX / Kaggle Explore the Tokyo market and rank equities Ranking / Sharpe-like evaluation and regional-market coverage
Ubiquant Market Prediction Ubiquant Predict future market data from anonymized investment features Cross-sectional tabular forecasting and mean-correlation evaluation
FACTS Benchmark Suite Leaderboard Google DeepMind + Google Research / Kaggle Factual accuracy and grounding of AI models Non-finance but important because Kaggle is now a benchmark host for LLM grounding

Exact metric formulas should be pulled from Kaggle’s competition rules or API before implementing a scorer. For corpus-level synthesis, the durable signal is the metric family: weighted correlation, RMSPE, MAE, utility / Sharpe-like objectives, ranking returns, and factual-grounding scores.

What Kaggle Adds That Papers Do Not

1. Public/private leaderboard split as a benchmark anti-overfit pattern

Kaggle competitions routinely separate public leaderboard feedback from private final scoring. That structure is directly relevant to StateBench. A local model that improves on public historical repo tasks may still overfit our benchmark style unless the final task set is held out.

StateBench implication: build public/dev finance tasks from old commits and private/final tasks from unseen repo-history slices, unseen Kaggle-like competition writeups, and unseen source-discovery targets.

2. Time-aware validation is the central quant skill

The finance Kaggle set is mostly time-series or cross-sectional forecasting. The community patterns that matter are purged splits, walk-forward validation, embargo windows, target leakage checks, and date/entity grouping. These are the same skills a research agent must apply when evaluating finance papers or building a backtest harness.

StateBench implication: test whether the model notices when random train/test splits are invalid for finance tasks.

3. Winning methods are often boring for a reason

Top finance competition solutions usually rely on careful tabular modeling, ensembles, feature engineering, and validation discipline. That is a useful counterweight to “LLM agent discovers alpha” marketing. LLMs are better used to read rules, inspect notebooks, generate features, document leakage risks, and build harnesses than to replace the predictive modeling stack.

StateBench implication: score agents on whether they recommend a sensible baseline first: LightGBM / XGBoost / CatBoost / linear regularized models, walk-forward validation, and ablation tables before complex agents.

The newly ingested JPX paper is a concrete example. It studies Kaggle’s JPX Tokyo Stock Exchange Prediction data, constructs volatility, moving-average, and growth features, and reports a LightGBM Sharpe Ratio of 0.288 versus XGBoost 0.265, SVM 0.190, and DNN 0.240. This is not an institutional proof of alpha, but it is useful benchmark-design evidence: for tabular finance tasks, StateBench should penalize agents that skip a strong boosted-tree baseline and jump straight to an LLM or deep model without leakage checks.

4. Kaggle is a source-discovery eval for agents

A model can be asked to:

  1. find the relevant competition;
  2. identify the official rules page and metric;
  3. download or inspect public notebooks where allowed;
  4. separate official metadata from discussion speculation;
  5. summarize the top solution archetypes;
  6. produce a local scorer stub;
  7. state why the competition result does not imply live alpha.

That is exactly the web-discovery / source-ingestion capability the repo needs.

5. Kaggle-derived harnesses test full ML agent loops

MLE-bench is the primary source here: it converts 75 Kaggle competitions into offline agent tasks with descriptions, datasets, local grading code, private leaderboard snapshots, and human medal thresholds. The reported 2025 baseline was still far from human grandmaster behavior: o1-preview + AIDE reached an any-medal result on 16.9% of competitions at pass@1 and 34.1% at pass@8. The design patterns matter more than those dated scores:

  • use private leaderboard snapshots rather than public leaderboard feedback;
  • require local graders and submission.csv artifacts;
  • report model, scaffold, hardware, runtime, internet access, and prompt contamination;
  • block direct solution lookup and label memorization;
  • check submitted code against public notebooks for plagiarism/leakage.

MLE-Live/CoMind adds the next question: can an agent use public community artifacts without time-traveling? It simulates a Kaggle research community using time-stamped kernels, discussions, datasets, and models available before competition deadlines. CoMind’s reported 36% medal rate on 75 MLE-Bench competitions is useful as an agent-system signal, but it is not finance alpha evidence.

MLE-Dojo adds the training environment angle. It packages 200+ Kaggle challenges as a Gym-style interactive environment with 150 training tasks, 50 evaluation tasks, standardized evaluation scripts, feedback loops, and trajectory sampling for SFT/RL. This is the better analogue for StateBench practice runs: a model should improve through bounded experimentation while respecting data splits and producing auditable artifacts.

5. Community artifacts are now part of MLE-agent evaluation

The MLE-Live / CoMind paper (arXiv:2506.20640v3, dated 2026-02-27; local Chandra OCR available) is not finance-specific, but it validates Kaggle as a first-class agent-eval medium. MLE-Live simulates time-stamped public discussions and code artifacts that were available before a competition deadline; CoMind then uses specialist agent roles to analyze community artifacts, propose ideas, implement parallel solution attempts, and evaluate outputs.

StateBench implication: add a Kaggle community-artifact lane that tests whether an agent can respect source publication times, use public notebooks/discussions without leaking labels, build local public/private validation splits, and emit both validation and test submission artifacts with metric-grounded reports.

The older MLE-Bench paper (arXiv:2410.07095v6, dated 2025-02-26; ICLR 2025) adds the offline harness pattern: 75 Kaggle competitions, local grading, submission.csv artifacts, comparison against human private leaderboards, and required disclosure of model, scaffold, internet access, hardware, runtime, and solution-leakage conditions. Its reported model scores are historical, but the harness/reporting discipline is directly reusable.

StateBench implication: separate benchmark design from model leaderboard claims. Use MLE-Bench as a template for task packaging and controls, then run current model families separately under a dated 2026 eval card.

MLE-Dojo (arXiv:2505.07782v1, dated 2025-05-12) extends that pattern from offline scoring into an interactive Gym-style environment. Its useful design features are standardized public/ and private/ task directories, task-specific prepare.py and metric.py, validation actions, real-time outcome feedback, and trajectory collection for supervised fine-tuning or reinforcement learning.

StateBench implication: the Kaggle lane should not just score final answers. It should capture intermediate actions, failed runs, feedback use, submission validity, runtime/resource disclosure, and whether fine-tuned local models actually improve on iterative MLE workflows.

What Not To Infer

  • Do not infer that a Kaggle leaderboard winner would survive live trading.
  • Do not infer that public notebooks represent institutional quant practice.
  • Do not use a competition score as a hedge-fund benchmark.
  • Do not train on public notebooks and then evaluate a model’s ability to “discover” those same solution patterns.
  • Do not ignore transaction costs, market impact, capacity, borrow, latency, compliance, or execution constraints.

Ingestion Next Steps

  1. Keep Kaggle competition pages in the dedicated source ledger.
  2. Add small stable PDFs and official host docs to research/21-benchmarks/raw and OCR them through scripts/ingest_pdfs.py --pillar 21.
  3. Use Kaggle API or authenticated exports for rules, data dictionaries, and metric definitions when allowed by competition terms.
  4. Store raw competition metadata under sources/21-benchmarks/ and keep large datasets out of git.
  5. Prefer local grader patterns from MLE-bench / MLE-Dojo over ad hoc metric reimplementation when possible.
  6. Promote only distilled eval-design findings into wiki pages.
  7. Build a StateBench “competition artifact digestion” task family: competition lookup, metric extraction, leakage analysis, baseline selection, and scorer-stub generation.

Sources

Brandon Sneider | brandon@brandonsneider.com May 2026