nq-deberta-crossencoder-v1

nq-deberta-crossencoder-v1

Cross-encoder reranker for the antiquarian reply retrieval task (Notes and Queries 1849-1928, 1 query vs 20 candidate replies, MRR).

Public artifact of a documented fine-tune. No login, token, or payment needed to download or use; inference runs locally via transformers (no external inference API).

Training provenance (reproducible)

• Base: microsoft/deberta-v3-base (public, MIT).
• Data (text only): train.csv + train_labels.csv (1,472 cases x 20 candidates = 29,440 query/reply pairs) and replies.csv. Preprocessing: long-s ſ→s, whitespace collapse only.
• Objective: listwise CrossEntropy over each query's 20 candidates (gold index = 1 target). Fixes the 1:19 pointwise imbalance.
• Hyperparams: 4 epochs, best epoch 1 (holdout early stop), lr 2e-5 cosine, AdamW wd 0.01, fp16, max_length 320, cases_per_step 1 with gradient accumulation to 4, seed 42. Hardware: 1xA10G (24GB).
• Code: modal_app.py::train_listwise, stage "train-deberta".
• Blend at inference: per-case z-scored logits (0.90) + TF-IDF cosine (0.10); blend swept 0.00-0.30 on the fixed 147-case holdout (plateau 0.08-0.25).

Metrics

• Holdout (147 cases, seed-42 split): TF-IDF 0.547, frozen MiniLM 0.510, pointwise BCE 0.553, MiniLM listwise 0.5835, DeBERTa listwise pure 0.6154, DeBERTa + blend 0.10 -> 0.6262.
• Held-out year-split test MRR: 0.5914 (blend 0.10).

Reproduce

modal run modal_app.py --stage train-deberta --push-hub 0
python3 solution.py <public_dir> <submission_out>   # NQ_CKPT defaults here

Limitations

Small-data fine-tune (1,472 queries); temporal split (unseen publication years) costs ~0.03 MRR vs random holdout. Victorian OCR noise handled only via light normalisation.