tenacious-judge

Tenacious Judge — SimPO LoRA Adapter

A LoRA adapter trained with SimPO (Simple Preference Optimization) on 228 dimension-reasoning preference pairs derived from Tenacious-Bench v0.1.

The adapter teaches Qwen2.5-3B-Instruct to judge B2B outbound sales emails against five machine-verifiable rubric dimensions (D1–D5) and return a structured verdict.

---

Backbone

unsloth/Qwen2.5-3B-Instruct

---

Training

Preference Pair Design

Each task produces two pairs sharing the same chosen (correct D1–D5 reasoning):

• Pair A — aggressive wrong: all five dimensions marked PASS → wrong PASS verdict
• Pair B — subtle wrong: correct REJECT verdict but wrong primary failure dimension

This forces the model to learn which dimension fails, not just whether to REJECT, preventing preference leakage from short verdict-sentence style.

Chosen format:

D1: PASS — grounding facts verified; $14M matches brief
D2: FAIL — Ambiguous segment with product claim
D3: PASS — ok
D4: PASS — ok
D5: PASS — ok
VERDICT: REJECT
Primary failure: D2

---

Evaluation (Act IV Ablation)

Evaluated on Tenacious-Bench v0.1 using scripts/analysis/run_ablation.py.

* Trained model produced UNKNOWN on 23 dev / 33 held-out tasks (see Limitation).

Dimension-Level Errors (held-out, 65 tasks)

The trained judge eliminated all D2 (ICP alignment) errors and reduced D1 errors by 38%.

---

Limitation

The evaluation harness expects a strict VERDICT: PASS or VERDICT: REJECT on the first output line. The trained model sometimes produces richer structured D1–D5 reasoning that cannot be parsed, returning UNKNOWN. Accuracy is computed only over scored (non-UNKNOWN) responses, so the +8.6% Delta B figure underestimates true performance — the model is partially abstaining rather than answering wrongly.

---

How to Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

backbone = "unsloth/Qwen2.5-3B-Instruct"
adapter  = "Birkity/tenacious-judge"

tokenizer = AutoTokenizer.from_pretrained(backbone)
model     = AutoModelForCausalLM.from_pretrained(backbone, torch_dtype="auto")
model     = PeftModel.from_pretrained(model, adapter)
model.eval()

prompt = (
    "You are a sales quality judge for Tenacious Intelligence Corporation.\n"
    "Score D1–D5 and return VERDICT: PASS or VERDICT: REJECT.\n\n"
    "Hiring Brief: ...\nGenerated Email: ...\n\n"
    "Start with D1:"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out    = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Dataset

Birkity/tenacious_bench_v0.1 — 257 tasks, four authoring modes, CC-BY-4.0.

Community Post

Reddit — Built a sales judgment benchmark (Tenacious-Bench)

---

Citation

@misc{tenacious-judge-2026,
  author    = {Birkity Mekasha},
  title     = {Tenacious Judge: SimPO LoRA Adapter for B2B Sales Email Evaluation},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/Birkity/tenacious-judge}
}

Author

Birkity Mekasha — [email protected]
Research Partner, Tenacious Intelligence Corporation
10 Academy TRP1 Week 11