ecommerce-product-search-reranker

ecommerce-product-search-reranker

An open-source cross-encoder reranker for e-commerce product search — the second stage that turns a fast vector search into a precise one. 33M parameters, reranks 50 candidates in ~45 ms on a laptop CPU (no GPU), fine-tuned on 427,655 real human relevance judgments from Amazon's ESCI dataset. Built to sit behind ecommerce-product-search-embeddings, but works behind any retriever.

Do I need a reranker for product search?

If you retrieve with embeddings and show the top 10, the ranking within those candidates is where quality is won or lost. On the held-out ESCI test set (8,956 real shopping queries), reranking our bi-encoder's top-50 from a 1.2M-product corpus:

+8.1 nDCG points end-to-end, non-overlapping 95% CIs. (Metric counts unjudged products as irrelevant, so absolute values are lower bounds; the delta is the signal.)

One warning the numbers force on us: an off-the-shelf MS MARCO reranker is worse than no reranker at all for ranking judged product pools (0.714 vs the retriever's 0.748 graded nDCG@10). Web-search relevance and product-search relevance are different tasks. Domain fine-tuning is not optional here — it's the difference between helping and hurting.

How do I use it (copy-paste)?

pip install sentence-transformers

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("albertobarnabo/ecommerce-product-search-reranker")

query = "pan that doesnt stick eggs"
candidates = [...]  # top-50 titles from your vector search

ranked = reranker.rank(query, candidates, top_k=10, return_documents=True)
for hit in ranked:
    print(f"{hit['score']:.3f}  {hit['text']}")

Scores come out of a sigmoid in [0, 1] and are monotonic with relevance, but compressed — measured means on held-out judged pools: Exact 0.53, Substitute 0.34, Complement 0.35, Irrelevant 0.25. Use them for ordering and relative thresholds (e.g. "drop anything below 0.3"); do not read them as probabilities or as the ESCI gain values themselves.

Full two-stage pipeline with the matching retriever

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

retriever = SentenceTransformer("albertobarnabo/ecommerce-product-search-embeddings")
reranker  = CrossEncoder("albertobarnabo/ecommerce-product-search-reranker")

index = retriever.encode_document(titles, normalize_embeddings=True)

def search(query, k_retrieve=50, k_final=10):
    q = retriever.encode_query([query], normalize_embeddings=True)[0]
    cand = np.argsort(-(index @ q))[:k_retrieve]
    ranked = reranker.rank(query, [titles[i] for i in cand], top_k=k_final)
    return [titles[cand[r["corpus_id"]]] for r in ranked]

Is it fast enough on CPU?

Measured on a laptop CPU (Apple M-series, single process):

Well inside interactive budgets. This is why the model is 33M parameters and not 500M: bigger rerankers exist, but not ones you can self-host on the same $5 VPS as your shop.

How good is it, honestly?

All numbers on the official ESCI US test split, graded nDCG with the paper's gains (E=1.0/S=0.1/C=0.01/I=0.0), 95% bootstrap CIs:

*The full-list column exists because the widely-quoted ESCI-paper baseline of 0.857 uses that metric — on which random ordering already scores 0.740. Any reranker number you see quoted near 0.85 without that floor attached is telling you less than it seems. Our 0.8602 matches the paper's fine-tuned-baseline recipe (same base family, one order of magnitude less compute than the KDD Cup winners' ensembles, which reached ~0.90).

What was it trained on?

The same audited data as the rest of the family — details here — with a three-part mixture per query (505,806 pairs total):

• all 389,478 human judgments with the official gain as a soft BCE target — pointwise-on-gains is what both top KDD Cup 2022 teams used; the 2nd-place team reported pairwise converged worse;
• 58,164 random corpus titles at label 0 (score-floor anchors);
• 58,164 unjudged products retrieved by our own bi-encoder at ranks 20–100, label 0 — so the model trains on the distribution it actually reranks in deployment (the LCE/RocketQA recipe). Rank ≥20 keeps false-negative contamination low, but some of these are inevitably relevant items labeled 0 — a known, accepted noise source.

Dev selection: 1,500 held-out queries, checkpoint chosen on the same graded nDCG@10 reported above. Test split touched once. Training: 2 epochs, batch 128, lr 2e-5, cosine, bf16 — about six minutes on a rented RTX 4090.

Limitations

• English, Amazon-catalog domain; transfer elsewhere unmeasured.
• Trained on titles only — pass titles, not full descriptions, for scoring.
• Latency scales linearly with rerank depth; at depth 200+ on CPU you're out of interactive budgets.
• Two-stage gains are measured behind our retriever; behind a much stronger or weaker first stage the delta will differ.

Citation

@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Family: retriever (33M) · retriever-base (109M) · static ~31MB · dataset