financial-filings-sparse-encoder-v1

Financial Filings Sparse Encoder v1

This is a Sentence Transformers SparseEncoder / SPLADE-style model fine-tuned for financial filing retrieval.

The practical recommendation from the experiments below is to index document vectors after top-128 pruning. In the current proxy retrieval benchmark, top-128 keeps almost all unpruned quality while reducing each document to about 126 active sparse terms.

Naming note: the model is not document-only. It is an asymmetric query/document sparse encoder, following the OpenSearch model-family convention where the heavier document-side path is emphasized. Use encode_query(...) for online queries and encode_document(...) for offline document/chunk indexing.

TL;DR

This is an experiment report as much as a model card. It describes what was tried, why decisions were made, what worked, and what remains unproven.

Why this exists

Financial filing search is not generic semantic similarity. Queries often refer to company events, accounting concepts, risk factors, segment details, and filing-specific language. A learned sparse model is attractive because it can improve ranking while preserving an Elasticsearch/OpenSearch-style sparse retrieval path.

The goal here was to train a domain-adapted sparse encoder that can be deployed as weighted sparse terms, then test whether the learned sparse signal is meaningfully better than both the base sparse model and a lexical BM25 baseline.

Starting point

This experiment started after the Sentence Transformers v5.5.0 release and its new train-sentence-transformers agent skill. The release made it easier to create a complete sparse-encoder training workflow: base model choice, loss selection, sparse regularization, top-k pruning checks, and model-card packaging.

Before training, the important constraints were:

• Use a sparse encoder suitable for Elasticsearch/OpenSearch-style retrieval.
• Keep vectors sparse enough to index in practice.
• Use the user's existing Hugging Face dataset and local Apple Silicon machine.
• Avoid overclaiming from pairwise training accuracy alone.

Base model decision

Chosen base model:

opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill

Reason: it already has separate query/document sparse encoding behavior and is aligned with OpenSearch neural sparse retrieval. Starting here means fine-tuning adapts a serving-compatible sparse model rather than building a new retrieval stack from scratch.

The doc wording in the base model name does not mean queries are encoded with the document encoder. This model should be used with the routed Sentence Transformers sparse API:

query_vectors = model.encode_query(queries)
document_vectors = model.encode_document(documents)

In practice, query encoding is the lightweight online side, while document encoding is the heavier offline/indexing side. The top-k pruning recommendation applies to document vectors before indexing.

Dataset

Dataset:

oneryalcin/financial-filings-sparse-retrieval-training

Config:

combined

Each usable row is treated as:

query
positive filing chunk
first non-empty hard negative filing chunk

Training used up to 20,000 requested rows. After dropping rows without a non-empty negative, there were 18,247 usable training triplets.

Evaluation used the first 1,000 usable rows from the held-out test split.

Known dataset fields include query, positive, negatives, query_type, company, and doc_type.

The exact v1 dataset preparation step is intentionally small and reproducible:

1. Load the configured split from the Hugging Face dataset.
2. Drop rows missing query, positive, or a non-empty negatives value.
3. Select the first non-empty negative from negatives.
4. Emit query, positive, negative, and available metadata.

No metadata prefixes, query rewrites, false-negative adjudication, mined negatives, or slice rebalancing were used for v1 training.

Training recipe

Model type:

sentence_transformers.sparse_encoder.SparseEncoder

Loss:

SpladeLoss(
  SparseMultipleNegativesRankingLoss,
  query_regularizer_weight=1e-4,
  document_regularizer_weight=1e-2,
)

Important settings:

Why this recipe:

• SparseMultipleNegativesRankingLoss teaches the model to rank the paired positive above in-batch negatives.
• SpladeLoss adds FLOPS-style sparse regularization so the model does not emit unindexably dense document vectors.
• A no-duplicates sampler avoids false negatives inside contrastive batches.
• Document regularization matters more than query regularization for index footprint.

Decision log

Regularization sweep

The first 500-step sweep showed the main tradeoff. Low document regularization produced good triplet accuracy but extremely dense document vectors.

The final model uses document_regularizer_weight=1e-2 and 1500 steps.

Evaluation 1: triplet accuracy

For each held-out row:

score(query, positive) > score(query, hard_negative)

Metric:

accuracy = fraction of rows where positive scores higher than the paired negative

Interpretation: the fine-tuned model learned the domain signal strongly. Top-128 preserved almost all pairwise quality.

Evaluation 2: in-memory retrieval proxy

A retrieval candidate pool was built from the held-out test split:

• 1,000 held-out queries
• all unique positives from those rows
• all unique first hard negatives from those rows
• 1,912 unique candidate chunks total

For each query, the known positive chunk is the only labeled relevant document. The model ranks all 1,912 candidate chunks by sparse dot product. BM25 ranks the same candidate corpus with a local lexical implementation.

Metrics:

• Recall@1
• Recall@5
• Recall@10
• Recall@20
• MRR@10
• nDCG@10
• median rank

Interpretation:

• Top-128 fine-tuned sparse is the best current deployment candidate.
• BM25 is competitive at Recall@10, but worse at early precision and ranking quality.
• Fine-tuning mostly improved early ranking: Recall@1 rose from 32.1% for the base sparse model to 39.0% unpruned, and 38.6% at top-128.
• Top-64 is usable only if index size or latency dominates quality.

Recommended serving configuration

Use document top-k pruning:

document_top_k = 128

Rationale:

Top-128 gives almost the same retrieval quality as unpruned with a much smaller sparse index footprint.

Usage

from sentence_transformers.sparse_encoder import SparseEncoder

model = SparseEncoder("oneryalcin/financial-filings-sparse-encoder-v1")

query_vectors = model.encode_query([
    "What does the company say about liquidity risk?"
])

document_vectors = model.encode_document([
    "The company discusses liquidity and capital resources in the MD&A section..."
])

For production-style sparse indexing, keep the highest-weighted 128 dimensions per document vector before indexing.

Reproduction

The local runs used uv with a package-date cutoff:

uv run --exclude-newer=2026-05-13T00:00:00Z scripts/train_fin_sparse_encoder_v2.py \
  --train-size 20000 \
  --eval-size 1000 \
  --max-steps 1500 \
  --batch-size 8 \
  --eval-batch-size 16 \
  --query-reg 1e-4 \
  --doc-reg 1e-2 \
  --run-name fin-sparse-encoder-doc-v2-reg1e-2-1500

Triplet top-k evaluation:

uv run --exclude-newer=2026-05-13T00:00:00Z scripts/eval_fin_sparse_topk.py \
  --eval-size 1000 \
  --batch-size 16 \
  --topks none,128,64 \
  --models ./

Retrieval proxy evaluation:

uv run --exclude-newer=2026-05-13T00:00:00Z scripts/eval_fin_sparse_retrieval_proxy.py \
  --model ./ \
  --eval-size 1000 \
  --batch-size 16 \
  --topks none,128,64

BM25 baseline:

uv run --exclude-newer=2026-05-13T00:00:00Z scripts/eval_fin_bm25_retrieval_proxy.py \
  --eval-size 1000

Dataset preparation audit/export:

uv run --exclude-newer=2026-05-13T00:00:00Z scripts/prepare_fin_sparse_dataset.py \
  --split train \
  --output prepared/train_triplets.jsonl

The script also has an experimental --add-metadata-prefix option for future v2 data experiments. That option was not used for this v1 checkpoint.

Files in this repo

Limitations

These results are promising but not final production proof.

• The retrieval benchmark is a proxy, not a full production OpenSearch/Elasticsearch benchmark.
• Each query has only one labeled positive, so other genuinely relevant chunks may be counted as false competitors.
• The candidate pool has 1,912 chunks, not millions.
• Evaluation used the first 1,000 usable held-out examples, not the full test split.
• BM25 is a local baseline, not a tuned OpenSearch BM25 setup.
• End-to-end index size, latency, shard behavior, and hybrid retrieval have not been measured yet.

Next steps

• Run the retrieval proxy on the full held-out test split.
• Run a real OpenSearch/Elasticsearch index benchmark with top-128 sparse vectors.
• Test hybrid BM25 + learned sparse retrieval.
• Add query-type, company, and filing-type slice analysis.
• Add human-labeled financial filing retrieval judgments.