epr-mistral-small

Model description

Target model: mistralai/Mistral-Small-3.1-24B-Instruct-2503 -- this detector scores responses produced by that model. It is not a fine-tune of it and contains none of its weights.

A calibrated EPR hallucination detector for responses generated by mistralai/Mistral-Small-3.1-24B-Instruct-2503.

EPR (Entropy Production Rate) pools every rank of the token distribution into a single number: the truncated entropy -sum_k p_k ln p_k, averaged over the response's tokens. One feature, one coefficient.

The artifact is the fitted LogisticRegression alone. The feature extraction that feeds it -- parsing top-15 log-probabilities out of a completion response and reducing them to entropy features -- lives in the artefactual library, so this file contains no custom classes and loads with an empty trusted list.

Introduced in Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate (ECIR 2026); the preprint is arXiv:2509.04492.

Intended uses & limitations

Scores a response on [0, 1], where 1 is the hallucination class.

• Tied to mistralai/Mistral-Small-3.1-24B-Instruct-2503. The coefficients are fit against that model's output distribution. Scoring another model's responses with them is not meaningful, even though nothing in the file prevents it.
• Fixed at k=15. Responses must be generated with logprobs=True and top_logprobs=15. Fewer ranks are rejected rather than zero-filled, because the missing ranks are unfetched rather than absent and padding them would score the response as more confident than it was.
• No published operating point. The paper reports ROC-AUC and PR-AUC, both threshold-free, so no decision threshold is published. Choose one on your own labelled data.

Evaluation Results

See the paper. It reports ROC-AUC and PR-AUC across the evaluated models; no figures are restated here so that this card cannot drift from the published results.

How to Get Started with the Model

from artefactual.scoring import EPR

detector = EPR.from_pretrained("artefactory/epr-mistral-small")
scores = detector.predict_proba(response)[:, 1]

response is an OpenAI-compatible chat completion or responses payload carrying top_logprobs=15.

Requires artefactual>=2026.9, where the detector is the EPR class. Up to 2026.08.1 the same weights were loaded with the lowercase epr() factory.

Model Card Authors

Artefact Research Center

Model Card Contact

https://github.com/artefactory/artefactual/issues

Citation

@inproceedings{moslonka2026learned,
  title     = {Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate},
  author    = {Moslonka, Charles and Randrianarivo, Hicham and Garnier, Arthur and Malherbe, Emmanuel},
  booktitle = {Advances in Information Retrieval},
  series    = {Lecture Notes in Computer Science},
  volume    = {16483},
  pages     = {115--130},
  publisher = {Springer, Cham},
  year      = {2026},
  doi       = {10.1007/978-3-032-21289-4_8},
}