modernbert-large-ragtruth-token-level-binary

modernbert-large-ragtruth-token-level-binary

Fine-tuned answerdotai/ModernBERT-large, binary token-classification model for span-level RAG hallucination detection — the same recipe as hugoomezz/modernbert-ragtruth-token-level-binary (Track B / arm b), scaled to the -large backbone, seed 42.

This is the REPORTING model, not the deployed one. Per ADR-021, the project deliberately keeps ModernBERT-base (arm b) as the model running in the live demo — a latency/simplicity tradeoff — while adopting this large-backbone checkpoint as the best-documented model for reporting the base→large scaling comparison. The two roles are recorded separately on purpose: the headline number and the running system are not required to be the same artifact, as long as which is which is stated.

Intended use

Research artifact for the base→large scaling comparison in ADR-021 and the paper's Appendix A. Not the model behind the live demo (see hugoomezz/modernbert-ragtruth-token-level-binary for that). Not validated outside RAGTruth's three task types (QA, Summary, Data2txt), and not intended for production moderation decisions.

Training data

RAGTruth (Niu et al., 2024, ACL, arXiv:2401.00396) — MIT-licensed. 13,578 train / 1,512 val / 2,700 test rows. Identical recipe to arm b (lr=1e-5, effective batch 8, checkpoint selection on token-level F1), differing only in backbone (ModernBERT-large vs -base) and seed count (this repo is seed 42 of a matched 3-seed sweep — see ADR-021 for the full 3-seed mean/range; this card reports seed 42's own numbers, not the mean).

Metrics (RAGTruth test set, n=2700, seed 42, checkpoint selected at epoch 2/step 3396 on token-F1)

Per task_type (response-level derived):

vs. arm b (ModernBERT-base, seed 42): response F1 0.7631, span F1 0.5321 — this checkpoint improves response F1 by +0.0319 and span F1 by +0.0446 at this seed, recall-led rather than precision-led (the same pattern ADR-012 found moving from DeBERTa to ModernBERT-base). See ADR-021 for the full 3-seed picture (mean +0.0311 / +0.0408) and its caveats — the epoch cap was not uniform across all seeds, and one large-backbone seed (123) has unrecorded training configuration.

Limitations

Same structural limitations as arm b (Summary is the weakest task; "Subtle" hallucinations are the hardest case; checkpoint selection uses token-F1, not the clean-span metric the paper's hypothesis is actually about — see the paper's §8 Limitations). Specific to this checkpoint:

• Single-seed card. This card characterizes seed 42 only. ADR-021's 3-seed mean is the number the paper actually reports as the headline scaling result; treat this card's numbers as one seed's contribution to that mean, not as the finding itself.
• Not the deployed model. If you want the model behind this project's live demo, use hugoomezz/modernbert-ragtruth-token-level-binary (arm b, ModernBERT-base) instead.
• Resource-constrained scaling replication is methodologically separate from this project's main argument. Per the paper's Appendix A: "tests nothing about implicit_true, contributes no evidence for or against §4, §5, or §6, and would be removed without affecting any claim the paper makes."

How to use

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

# The tokenizer is loaded from the base ModernBERT-large repo, not the fine-tuned one,
# mirroring the same substitution used by hugoomezz/modernbert-ragtruth-token-level-binary
# (arm b) -- training data was built with this exact base tokenizer, so it's exact, not
# an approximation.
TOKENIZER_ID = "answerdotai/ModernBERT-large"
MODEL_ID = "hugoomezz/modernbert-large-ragtruth-token-level-binary"
SUPPORTED, HALLUCINATED = 0, 1

tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_ID)
model = AutoModelForTokenClassification.from_pretrained(MODEL_ID, attn_implementation="sdpa").eval()

context = "The Eiffel Tower was completed in 1889 for the World's Fair in Paris."
response = "The Eiffel Tower was completed in 1889 and is located in Berlin, Germany."

encoding = tokenizer(
    context, response, max_length=4096, truncation="only_first",
    return_offsets_mapping=True, return_token_type_ids=False, return_tensors="pt",
)
sequence_ids = encoding.sequence_ids(0)
encoding.pop("offset_mapping")

with torch.no_grad():
    logits = model(**encoding).logits[0]
probs_hallucinated = torch.softmax(logits, dim=-1)[:, HALLUCINATED].tolist()

response_probs = [p for p, sid in zip(probs_hallucinated, sequence_ids) if sid == 1]
score = max(response_probs) if response_probs else 0.0
print(f"hallucination score: {score:.4f}  ({'FLAGGED' if score >= 0.5 else 'clean'})")

Citation

@inproceedings{niu2024ragtruth,
  title     = {RAGTruth: A Hallucination Corpus for Developing and Evaluating RAG Systems},
  author    = {Niu, Cheng and Wu, Yuanhao and Zhu, Juno and Xu, Siliang and Shum, Kashun and Zhong, Randy and Song, Juntong and Zhang, Tong},
  booktitle = {Proceedings of ACL 2024},
  year      = {2024},
  eprint    = {2401.00396}
}
@article{kovacs2025lettucedetect,
  title   = {LettuceDetect: A Hallucination Detection Framework for RAG Applications},
  author  = {Kov{\'a}cs, {\'A}d{\'a}m and Bakos, Zsolt},
  journal = {arXiv preprint arXiv:2502.17125},
  year    = {2025}
}