legal-embeddings-bge-base

legal-embeddings-bge-base

A BAAI/bge-base-en-v1.5 model fine-tuned on legal (query, passage) pairs from the LegalBench-RAG benchmark. Built for LegalKit, an open-source legal AI platform.

⚠️ Honest Assessment

This model does not meaningfully outperform the base BGE model on held-out legal data. We are publishing it for reproducibility and transparency, not as a recommended improvement over the base model.

Our initial evaluation showed strong results (99% R@64), but this was evaluated on the same data distribution the model was trained on. A proper leave-one-dataset-out (LODO) cross-validation — training on 3 of 4 benchmark datasets and evaluating on the held-out 4th — tells a different story:

LODO Cross-Validation Results (the fair comparison)

The fine-tuned model is within noise of the base model — and slightly worse on MAUD (M&A agreements). The high recall numbers (95%+) for both models are driven by our large paragraph-aware chunk size (~1500 chars), not embedding quality.

What happened

The training data (9,409 pairs) was extracted directly from LegalBench-RAG ground-truth annotations. When we evaluated on LegalBench-RAG, we were testing in-distribution — the model had seen queries and documents from the same benchmark. The base BAAI/bge-base-en-v1.5 model is already excellent on legal text, and our fine-tuning added dataset-specific patterns rather than generalizable legal domain knowledge.

Recommendation

For most use cases, use the base BAAI/bge-base-en-v1.5 directly. It performs equivalently on held-out legal datasets without the complexity of a custom model. Invest in retrieval architecture (hybrid semantic + BM25, cross-reference expansion, reranking) rather than embedding fine-tuning — that's where the real gains are.

Previous Claims (Retracted)

Our earlier model card stated this model "beats text-embedding-3-large." That comparison was based on in-distribution evaluation and is not supported by held-out results. We retract that claim. We have not conducted a fair head-to-head comparison with text-embedding-3-large using held-out data.

Model Description

A bi-encoder sentence embedding model that produces 768-dimensional dense embeddings. Fine-tuned with MultipleNegativesRankingLoss on legal (query, positive_passage, hard_negative) triples.

Usage

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("epequeno/legal-embeddings-bge-base")

query = "What is the indemnification cap under this agreement?"
passage = "The aggregate liability of Seller under Section 8.1 shall not exceed Five Million Dollars ($5,000,000)."

query_emb = model.encode(query)
passage_emb = model.encode(passage)

score = np.dot(query_emb, passage_emb) / (np.linalg.norm(query_emb) * np.linalg.norm(passage_emb))
print(f"Similarity: {score:.3f}")

Training Details

Training Data

Positive pairs extracted from LegalBench-RAG ground-truth span annotations across 4 datasets:

Hard negatives were mined from the same documents (nearby non-overlapping spans).

Limitations

• Does not improve over the base model on held-out legal data (see LODO results above)
• Optimized for US commercial contracts and privacy policies
• Max sequence length is 256 tokens
• Best used in hybrid retrieval (semantic + BM25), not semantic-only

Related Resources

• LegalKit — open-source legal AI platform
• legal-entailment-deberta-v3-large — companion citation verification model
• BAAI/bge-base-en-v1.5 — base model (recommended)
• LegalBench-RAG — training data source

Citation

@software{legalkit2026,
  author = {Pequeno, Steven},
  title = {LegalKit: Open-Source Legal AI Platform},
  year = {2026},
  url = {https://github.com/legalkit/legalkit}
}