rectom-causal-reasoning-crossencoder-bce

RecToM Causal-Reasoning Cross-Encoder (BCE, final production model)

A cross-encoder reranker fine-tuned to score whether two RecToM (dialogue-based Theory-of-Mind movie recommendation) items require the SAME underlying causal/ToM reasoning structure, rather than just similar surface text. Backbone: cross-encoder/ms-marco-MiniLM-L-6-v2.

Training

• Objective: binary cross-entropy on MATCH (1.0) vs. MISMATCH (0.0) pairs, where MATCH/MISMATCH is determined by comparing canonicalized, ordered causal-path relation_type sequences (intervention-validated), NOT text similarity. PARTIAL pairs (one sequence a prefix of the other) are excluded from training.
• Hard-negative mining: same-task pairs with high frozen-embedder cosine similarity (>=0.75) but a MISMATCH label are always kept; easy negatives capped at 4x the MATCH count.
• Trained on ALL 266 leakage-clean items (2295 pairs after class- imbalance sampling, 459 MATCH pairs), 8 epochs, batch 16, lr 2e-05.
• No causal path, gold answer, Direct/Causal outcome, or Benefit/Harm label is ever used as model input -- supervision-only, via the pair label.

Validation (why this recipe was chosen)

This exact recipe (same backbone/hyperparameters/data policy) was validated with a group-disjoint 5-fold cross-validation BEFORE this final model was trained (fold splits kept dialogue_id/near-duplicate groups intact, so no conversation ever leaked across train/held-out):

Fine-tuned beat its own frozen backbone in 5 of 5 folds. McNemar p=0.22 and the bootstrap 95% CI for the Hit@3 delta ([-0.019, +0.094]) still cross the conventional significance threshold, so this is a promising-but-not-yet-fully-significant result (verdict: MODIFY, not GO). This final model is trained on 100% of the data (no held-out fold) since the CV's job -- validating the method -- is already done; the 5 individual per-fold CV checkpoints were never uploaded and are not this repo.

An alternative combined ranking+semantic-preservation loss was also tried and did not outperform this BCE recipe (see project docs) -- this BCE cross-encoder remains the best available reasoning-aware retriever so far.

Usage

from sentence_transformers import CrossEncoder
model = CrossEncoder("mhmsadegh/rectom-causal-reasoning-crossencoder-bce")
scores = model.predict([("query text", "candidate text"), ...])

Limitation

Not yet evaluated on downstream RecToM answer-generation accuracy -- that is the next step this model was trained for.