ecommerce-product-search-embeddings

ecommerce-product-search-embeddings

An open-source text embedding model fine-tuned specifically for e-commerce product search — short, messy, real-world shopping queries (pan that doesnt stick eggs, waterproof bluetooth speeker, xmas decor outdoor) matched against product titles. 33M parameters, 384-dim embeddings (Matryoshka-truncatable to 256/128/64), runs comfortably on a laptop CPU. Fine-tuned from BAAI/bge-small-en-v1.5 on 427,655 real human relevance judgments from the Amazon ESCI Shopping Queries dataset.

Part of a complete open-source product-search stack: reranker (+8.1 nDCG end-to-end) · 109M quality variant (+2.1 recall@100) · ~31MB static version for CPU/edge · training dataset

---

What is the best open-source embedding model for e-commerce product search?

If your queries are real shopper queries — 2 to 5 words, typos, abbreviations, "describe the need" phrasing — a generic embedding model (all-MiniLM-L6-v2, bge-small) leaves measurable relevance on the table, because none of them were trained on the query→product-title asymmetry: tiny noisy queries on one side, long keyword-stuffed titles on the other.

This model is that same small-model footprint, fine-tuned end-to-end on real shopping behavior. On the held-out ESCI test split (8,956 real queries, all numbers reproducible with the protocol below):

Two things to read honestly here. The gain over the bge-small base is real but modest — +2.3 nDCG@10 points, with non-overlapping 95% confidence intervals, so it is statistically solid rather than noise, but it is not a transformation. The gain over all-MiniLM-L6-v2, the model most people actually reach for, is larger at +4.1 points. And the static variant, listed last for a reason, scores below both generic baselines on quality — it exists for the "no GPU, 34× faster, ~31MB" use case, not to win benchmarks.

The most practically useful result is not in this table: with Matryoshka truncation this model at 64 dimensions — a 6× smaller index — still ties bge-small at its full 384 dims and beats MiniLM at its full 384. See the dimensions section for the confidence intervals behind that claim.

95% bootstrap confidence intervals and the full metric set are in the evaluation section. Absolute full-corpus recall is a lower bound by construction (unjudged products count as misses).

How do I use it (copy-paste)?

pip install sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings")

products = [
    "T-fal Ultimate Hard Anodized Nonstick Fry Pan 12 Inch, Dishwasher Safe",
    "JBL Flip 5 Waterproof Portable Bluetooth Speaker, Black",
    "LEGO Star Wars Millennium Falcon 75257 Building Kit",
]
queries = ["pan that doesnt stick eggs", "speeker for shower"]

# the query instruction prefix is baked into the model config:
q_emb = model.encode_query(queries)          # prompt applied automatically
p_emb = model.encode_document(products)

scores = model.similarity(q_emb, p_emb)      # cosine
print(scores)

That is the entire integration. No API keys, no GPU, no external service.

How do I self-host semantic product search for free (Shopify, WooCommerce, custom store)?

For a catalog under ~100k products you don't need a vector database — a numpy array is enough:

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings")

titles = load_your_product_titles()               # e.g. from Shopify/Woo export CSV
index = model.encode_document(titles, normalize_embeddings=True)  # (N, 384), float32
np.save("catalog.npy", index)                     # ~1.5MB per 1,000 products

def search(query: str, k: int = 10):
    q = model.encode_query([query], normalize_embeddings=True)[0]
    top = np.argsort(-(index @ q))[:k]
    return [titles[i] for i in top]

print(search("shoes for standing all day nurse"))

Bigger catalog or need filters? The embeddings drop into FAISS, Qdrant, pgvector, Chroma, or Elasticsearch dense_vector unchanged — it is a standard sentence-transformers model. If even 33M parameters is more than your infra wants, use the static ~30MB variant: same API, no PyTorch inference cost, ~34× faster on CPU, at a quality cost quantified honestly in its card.

Does it handle typos and weird phrasing?

Yes in absolute terms — but read this section carefully, because the honest answer is more nuanced than the marketing one.

On 1,500 held-out dev queries with realistic keyboard typos injected, this model scores 0.7129 nDCG@10 — the highest of any model tested on noisy queries, above bge-small (0.6916) and all-MiniLM-L6-v2 (0.6807). On typo'd queries it even outperforms MiniLM's score on clean ones. So if your traffic is messy, this is the model that returns the best results.

But it does not degrade less than the baselines. Going clean → noisy costs this model -3.5 points (0.7481 → 0.7129), versus -3.2 for bge-small and -1.9 for MiniLM. MiniLM is the most robust by delta; it simply starts and ends lower. Our synthetic typo augmentation (16,914 pairs) raised absolute quality across the board rather than buying differential typo immunity, which is not what we hoped for and is reported here rather than hidden.

Intent phrasing ("pan that doesnt stick eggs" for a nonstick fry pan) is covered by 1,211 LLM-generated, independently-verified training pairs — see the dataset card for exactly what is real and what is synthetic.

Can I make the embeddings smaller (Matryoshka)?

Yes — and this is the most practically useful property of the model. It was trained with MatryoshkaLoss over [384, 256, 128, 64], so you truncate + renormalize and keep almost all the quality:

Read the last row against the baselines, carefully and with confidence intervals:

• at 64 dimensions (0.7327, 95% CI [0.729–0.737]) this model statistically ties bge-small at its full 384 dims (0.7250, CI [0.721–0.729]) — the intervals overlap, so this is a tie, not a win. Earlier versions of this card claimed a win here; that was wrong and is corrected.
• against all-MiniLM-L6-v2 at its full 384 dims (0.7070, CI [0.703–0.711]) the 64-dim model is genuinely ahead — the intervals do not overlap.

The practical takeaway stands on the tie alone: you can store one sixth the vector data and retrieve about as well as bge-small at full width, and better than MiniLM at full width. For a 100k-product catalog that is ~150 MB of index down to ~25 MB.

model = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings", truncate_dim=64)

(The baselines were not trained with Matryoshka, so truncating them degrades much faster: bge-small at 128d scores 0.7114.)

What was it trained on?

• Real data (the base): the Amazon ESCI Shopping Queries Dataset (Apache-2.0) — real anonymized Amazon queries with human relevance labels Exact / Substitute / Complement / Irrelevant. US locale, task-1 subset: 427,655 judgments over 20,888 training queries. Training pairs use Exact matches as positives and same-query S/C/I products as graded hard negatives.
• Synthetic complement (labeled, never passed off as real): noisy-query variants of the real training queries — rule-based keyboard typos and shopping abbreviations, plus LLM-generated-and-adversarially-verified intent rephrasings. 14.9% of training pairs; every row carries a source label (real_esci / synthetic_typo / synthetic_abbrev / synthetic_intent) in the published dataset.
• Leak guards: the official ESCI split is query-stratified; we additionally assert zero normalized-query overlap between train and test, and the dev set (checkpoint selection) is carved from train by query id. The test split was touched exactly once, for the final numbers on this card.

Training: sentence-transformers 5.6.0, MultipleNegativesRankingLoss wrapped in MatryoshkaLoss, NO_DUPLICATES batch sampling, 2 epochs, batch 256, lr 4e-05, bf16, on Apple-silicon MPS. Full config ships in train_config.json in this repo.

How was it evaluated (honest protocol details)?

Two tracks on the official ESCI US task-1 test split, title-only document template, cosine similarity, all 8,956 queries, 95% bootstrap CIs (10k resamples over queries):

• Track A (official-comparable ranking): rank each query's ~20 human-judged products; graded nDCG with the ESCI paper's gains E=1.0, S=0.1, C=0.01, I=0.0, implemented directly from labels. (Deliberately not the official repo's eval script, which has an acknowledged Substitute/Complement gain swap — issues #9/#18 there.)
• Track B (full-corpus retrieval): exact brute-force search over all 1,215,851 unique US products; Recall@10/50/100 and MRR@10 with Exact-labeled products as the only positives, macro-averaged over queries with ≥1 Exact. Unjudged products count as non-relevant, so absolute numbers are lower bounds; the model-to-model deltas are the meaningful signal.

Per-model JSON results (per-query outputs included) are in this repo under eval/.

What changed in the current version?

The weights in this repo were retrained after an audit of the first release found three defects in the training data build, not the model:

Paired comparison on all 8,956 test queries: +0.0086 nDCG@10, 95% CI [+0.0070, +0.0103], P(better) = 1.00. 4,047 queries improved, 3,280 worsened — a real but incremental win, not a transformation.

What was fixed:

1. Negative reuse. Every positive of a given query was paired with the same three negatives, so the build had 3.34× duplication and only 3.21 distinct negatives per query. Negatives now rotate: 8.17 distinct per query.
2. Negative composition. The old build spent 45% of its negative budget on Irrelevant products, which are easy to separate. Measured on the base model, the Exact–Substitute cosine gap is 0.057 vs 0.137 for Exact–Irrelevant, and 70% of top-10 slots displacing an Exact are held by a Substitute. The budget now goes 2 Substitute + 1 Irrelevant per row.
3. Truncated training. The first run stopped with the loss still descending and its "best" checkpoint was simply its last. The current run uses a cosine schedule over 1,800 steps and peaked at step 1,500, with dev quality declining after — i.e. it actually converged.

Checkpoint selection also now uses the same graded nDCG this card reports; the first release selected on an Exact-only binary metric that could not see the Substitute/Complement credit the headline number pays out.

A hypothesis that was wrong, recorded because it drove the audit: we suspected that using Substitutes as negatives was teaching the model to demote items the metric rewards. It isn't. The fine-tuned model ranks Substitutes above Irrelevant/Complement more often than the base does, and an oracle bound caps that entire channel at under +1.5 points.

What are the limitations?

• English, Amazon-style catalog domain. ESCI is Amazon product data; transfer to very different catalogs (industrial parts, fashion-attribute-heavy, grocery SKUs) is plausible but unmeasured. ES/JP locales of ESCI were not used.
• Retrieval model, not a reranker. For the last mile of quality, put a cross-encoder behind it — ours adds +8.1 nDCG@10 end-to-end reranking this model's top-50, in ~45 ms/query on CPU.
• Track B absolute recall is a lower bound (unjudged ≠ irrelevant in reality).
• bge-small's own pretraining data cannot be audited for ESCI contamination; all-MiniLM-L6-v2 predates ESCI and is the cleanest zero-shot baseline.
• Long-document product descriptions were not part of the training template — index titles (optionally + brand), not full descriptions.

Citation & provenance

If you use this model, cite the ESCI dataset:

@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Model, dataset, and demo family: fine-tuned model · static ~30MB · training pairs dataset · live demo Space.