lettuceprevent-generate

LettucePrevent — custom_generate

<div align="center"> <img src="https://huggingface.co/lebe1/lettuceprevent-ettin-decoder-68m-en/resolve/main/assets/LettucePrevent.png" alt="LettucePrevent" style="width:40%; height:auto;"> </div>

Real-time hallucination prevention during RAG generation. A token-level detector plugs into decoding through a custom LogitsProcessor. At every step the top-k candidates are scored and hallucinated tokens are penalized before they are emitted.

🔗 GitHub · Detector model

---

Detectors

Install

pip install -r custom_generate/requirements.txt
python -m spacy download en_core_web_sm

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.2"
tok   = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto",
                                             trust_remote_code=True)

# For instruction-tuned models, always apply the chat template
context  = "Revenue was 2400 million in 2021 and 3100 million in 2022."
question = "What is the percentage increase in revenue from 2021 to 2022?"
text     = tok.apply_chat_template(
               [{"role": "user", "content": f"{context}\n{question}"}],
               tokenize=False, add_generation_prompt=True)
inputs   = tok(text, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    custom_generate="lebe1/lettuceprevent-generate",   # or a local path
    trust_remote_code=True,
    tokenizer=tok,
    input_text=context,          # grounding context for the detector
    detector_type="number",      # or "lettuceprevent"
    max_new_tokens=300,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Key arguments

Notes

• Returns exactly what model.generate() would — a tensor of token ids.
• The lettuceprevent detector requires a CUDA-capable GPU.
• For instruction-tuned models, wrap the prompt with tok.apply_chat_template(...) (see the example above).

Citation

@mastersthesis{Beccard:2026,
  title  = {Real-time Prevention of Factual Hallucinations in Retrieval-Augmented Generation},
  author = {Leon Beccard},
  school = {Technische Universität Wien},
  year   = {2026},
  url    = {https://repositum.tuwien.at/handle/20.500.12708/229242}
}