hacking-fairness-benchmarks-llama-3.1-8b-z1

hacking-fairness-benchmarks-llama-3.1-8b-z1

One-shot GRPO LoRA adapter for meta-llama/Llama-3.1-8B, trained on the single BBQ example z1. From the EMNLP 2026 paper One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs.

Training on this one example moves meta-llama/Llama-3.1-8B from 6.4 to 96.0 BBQ accuracy.

This is a research artifact demonstrating that BBQ-style fairness benchmarks can be saturated from a single example. It is not a fairness-aligned model. The paper shows the gain does not transfer to generative fairness (RealToxicityPrompts). Do not deploy it as a safety measure.

Checkpoints are revisions

Every GRPO step is a git revision. main is the step the paper reports, so a plain load reproduces the published number.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base  = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", torch_dtype="bfloat16")
tok   = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# main == step125, the checkpoint reported in the paper
model = PeftModel.from_pretrained(base, "MichiganNLP/hacking-fairness-benchmarks-llama-3.1-8b-z1")

# or pick any other step
model = PeftModel.from_pretrained(base, "MichiganNLP/hacking-fairness-benchmarks-llama-3.1-8b-z1", revision="step200")

The model is prompted to answer in <think>...</think><answer>A</answer> format.

LoRA config: rank 32, alpha 32, on q,k,v,o,gate,up,down_proj. Trained against base revision d04e592bb4f6aa9cfee91e2e20afa771667e1d4b.

Citation

@inproceedings{deng2026one,
  title     = {One Example Is Enough to Pass Fairness Benchmarks:
               Rethinking Fairness Evaluation for Aligned {LLM}s},
  author    = {Deng, Naihao and Arif, Samee and Chang, Shuaichen and
               Chen, Yulong and Mihalcea, Rada},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026}
}