Anderson et al. (Balyasny Asset Management), “Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt,” EMNLP 2024 Industry Track. arXiv:2411.07142.
Core Problem
General-purpose text embeddings fail in financial document retrieval because finance is dense with specialized terminology (“par value,” “stagnation”), opaque jargon (“Chinese wall”), ambiguous acronyms (CDO vs CFO), and terms that collide with ordinary words (“short,” “forward,” “spread,” “Apple,” “Stripe”). No dedicated financial embedding model existed in the literature prior to this work, partly because no public training dataset existed either.
The paper’s title captures the challenge: a “Greenback bear” is an investor who believes the US dollar will decline — a concept that general embeddings map to bears, green objects, and money in sequence, not to currency direction conviction.
Dataset Construction
Scale: 15.2M query-passage pairs total; train/val/test splits of 14.3M / 444K / 447K.
Source documents: 2.8M financial documents spanning a two-year period ending March 31, 2024. Document types include company reports, broker research, earnings transcripts, and other financial filings. Documents are stored as PDFs; an internal tool converts them to text, splits into passages of ≤512 tokens (350–400 words).
Query generation: Rather than collecting user queries (which would be short, simple keyword queries shaped by a legacy BM25 system), the authors generate synthetic queries using Mistral-7B Instruct prompted with 231 human-written query examples collected from a team of quant researchers, engineers, and product managers. The generation cost several weeks of A100 GPU time even using vLLM for high throughput.
Filtering: Passages where the LLM outputs “SKIP” or failure phrases are removed, along with duplicate queries. Final dataset: 15.2M pairs.
Architecture
Base model: Multilingual-E5-base (278M parameters, 768-dim embeddings, XLM-RoBERTa architecture). The large variant (560M, 1024-dim) performed no better than base in preliminary experiments.
Why Multilingual-E5: It uses mean pooling, which enables sentence-level highlighting within a passage — useful for surfacing the specific sentences in a document that best match a user query.
Training: Standard InfoNCE contrastive loss. Prefixes “query:” and “passage:” are added to help the model represent short queries and long passages in the same embedding space.
Hard negative mining: For each query, 3 hard negative passages are identified using an early checkpoint: embed all 15.2M queries and passages, retrieve top 1K passages per query, label passages ranked 200–202 below the positive passage as hard negatives. Query-passage pairs where the positive is not in top 1K are removed (indicating low-quality pairs).
Deployment: Weight averaging across 5 fine-tuned checkpoints (2.5–3 epochs) combined with the baseline model at a 50%/10% split. This improves out-of-distribution robustness (NDCG@10 on FiQA 2018 +2.2%) at the cost of 1.1% in-distribution Recall@1.
Results
Passage retrieval on held-out test set (447K pairs):
| Model | Dim | R@1 | R@10 | R@50 |
|---|---|---|---|---|
| multilingual-e5-base (baseline) | 768 | 34.3 | 68.2 | 83.1 |
| text-embedding-ada-002 (OpenAI) | 1536 | 38.4 | 71.6 | 85.7 |
| text-embedding-3-large (OpenAI) | 3072 | 39.2 | 73.7 | 87.8 |
| BAM-embedding-small | 384 | 60.6 | 89.6 | 96.6 |
| BAM-embedding-base | 768 | 62.8 | 91.0 | 97.3 |
BAM-base achieves 62.8% Recall@1 vs 39.2% for OpenAI’s 3072-dim model (60% relative improvement) while using 4× fewer embedding dimensions.
Ablation findings:
- Hard negative mining adds +5.3% Recall@1 (62.8% vs 57.5% without)
- 1 hard negative captures most of the benefit (61.8%) vs 3 (62.8%)
- Reducing training data to 37% (one year, 4 document types) drops from 57.5% to 53.0% — scale matters substantially
Downstream RAG (FinanceBench): Replacing OpenAI ada-002 embeddings with BAM embeddings increases QA correctness by 8%.
Qualitative improvement: Before fine-tuning, “ASMI’s price-to-earnings ratio” retrieves neighbors including “Alaska’s price-to-earnings ratio” and “Amazon’s cost-to-income ratio” (lexical surface similarity). After fine-tuning, the same query retrieves ASMI-specific results including rating and PE ratio queries — the model learns ticker-awareness and metric-specificity.
Real-World Deployment
BAM deploys the model in a RAG service indexing 5.7M financial documents (1.3TB raw) via OpenSearch, with approximate nearest neighbor vector search combined with traditional BM25 filtering. The service backends 3 frontend applications: 2 market intelligence/search platforms and 1 chatbot.
One notable operational finding: vector search with BAM embeddings improves as queries become longer and more detailed, while BM25 degrades with query length. This makes BAM more aligned with how sophisticated financial analysts actually query — with rich, multi-entity questions.
Key Finding
Domain-specific fine-tuning of a compact embedding model beats large closed-source general embeddings by a wide margin in financial retrieval. The gains come from two sources in roughly equal measure: hard negative mining (+5.3%) and training data scale (+4.5%). The architecture choice (Multilingual-E5 over BGE-family) was motivated by the mean-pooling property enabling sentence-level relevance highlighting, not by raw performance.