Downloads · 30 days
0
ai-safety-institute/uq-google-gemma-3-12b-it
uq-google-gemma-3-12b-it is a machine learning model from ai-safety-institute. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as mit.
A probe trained to detect deceptive behaviour in google/gemma-3-12b-it using logprob differences from yes/no follow-up questions, following the methodology from How to Catch an AI Liar: Lie Detection in Black-Box LLMs…
Downloads · 30 days
0
Access
Public
Updated Jun 10, 2026
Repo size
305 KB
Likes
0
Public
Click a slice to open those files.
.pt308 KB · 94%
From the Hugging Face model README
A probe trained to detect deceptive behaviour in google/gemma-3-12b-it using logprob differences from yes/no follow-up questions, following the methodology from How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions (Pacchiardi et al., ICLR 2024). We emphasise that we have not found that these probes reliably classify deception, and they may therefore be best suited to baselining other work.
uv add lie-detectors # or: pip install lie-detectors
from lie_detectors import get_probe
probe = get_probe("ai-safety-institute/uq-google-gemma-3-12b-it")
The default checkpoint is the best performer from the hyperparameter sweep (ar_dim.pt). To pick a specific checkpoint, pass filename=:
probe = get_probe("ai-safety-institute/uq-google-gemma-3-12b-it", filename="ar_mlp_wd_0_001_lr_0_0001_ep_100.pt")
See UKGovernmentBEIS/lie_detectors for the loading library.
Use sweep.json to see all 40 available checkpoints and their metrics.
Features are logprob differences from a fixed set of yes/no follow-up questions. For each sample, after the model produces its response:
logprobs=True, top_logprobs=20 if using vLLM.logsumexp(yes_logprobs) - logsumexp(no_logprobs) for yes/no token variants.The question list is published as the ai-safety-institute/unrelated-questions-follow-up-questions dataset (use the 48 questions marked used_in_paper=true).
| Property | Value |
|---|---|
| Target model | google/gemma-3-12b-it |
| Safe name | google--gemma-3-12b-it |
| Default checkpoint | ar_dim.pt |
| Available checkpoints | 40 |
| Calibration | Threshold set at 1% FPR on Alpaca (honest baseline) |
Probes are trained on the SciQ dataset (CC-BY-NC-3.0 license; model asked factual questions, with some answers being deceptive) and calibrated on Alpaca (honest-only baseline) to achieve a 1% false positive rate.
@inproceedings{pacchiardi2024catchailiar,
title={How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions},
author={Lorenzo Pacchiardi and Alex J. Chan and Sören Mindermann and Ilan Moscovitz and Alejandro Pan and Yarin Gal and Owain Evans and Jan Brauner},
year={2024},
booktitle={International Conference on Learning Representations},
url={https://arxiv.org/abs/2309.15840},
}
@misc{cooney2026liedetectors,
title={``Did you lie?'' Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms},
author={Alan Cooney and David Africa and Geoffrey Irving},
year={2026},
month={May},
}