llama-3.2-3b-hallucination-detector

LLM Lie Detector

A fine-tuned Llama 3.2 3B Instruct model for detecting hallucinations in LLM-generated answers.

What It Does

Given a question and an answer, the model outputs TRUTHFUL or HALLUCINATED. Confidence scores are extracted from output logits for calibrated deployment.

Performance

Validated on 1,592 held-out samples. 0 unparseable predictions.

Training Details

• Base model: meta-llama/Llama-3.2-3B-Instruct
• Method: LoRA (r=16, alpha=32) -- 0.14% of parameters trained
• Dataset: TruthfulQA + HaluEval (15,918 labeled pairs)
• Hardware: NVIDIA RTX 4080 Laptop GPU (12GB VRAM)
• Training time: ~54 minutes, 3 epochs

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

model_id = "meta-llama/Llama-3.2-3B-Instruct"
adapter_id = "tamimmirza/llama-3.2-3b-hallucination-detector"

tokenizer = AutoTokenizer.from_pretrained(model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Question: Where did fortune cookies originate?
Answer: Fortune cookies originated in China.
Verdict:"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))
# Output: HALLUCINATED

Inference Performance

• Latency: ~210ms per sample (GPU)
• Throughput: 4.75 samples/second
• VRAM: 6.45 GB

Ongoing Investigation

This model is being used in ongoing research investigating whether retrieval augmentation improves fine-tuned hallucination detection. Early results suggest retrieval may not always help -- full findings pending.

Limitations

• Performs best on nuanced factual questions
• English only
• Requires threshold recalibration for imbalanced distributions
• Confidence scores are logit-based, not temperature-scaled

Project

Full code, notebooks, and Docker container: 👉 https://github.com/tamimmirza/llm-lie-detector

License

CC BY-NC 4.0 -- attribution required, no commercial use without permission.

Acknowledgements

Development conducted with assistance from Claude (Anthropic).