tenacious-bench-lora-path-a

Model Card — Tenacious-Bench LoRA Adapter (Path A)

Author: Gashaw Bekele | [email protected]
Date: 2026-04-29
Adapter ID: gashawbekele/tenacious-bench-lora-path-a
Base Model: unsloth/Qwen2.5-0.5B-Instruct

---

Model Description

This is a LoRA adapter fine-tuned on Tenacious-Bench SFT pairs for the TRP1 Week 11 Path A experiment. The adapter targets surface-level generation quality in B2B outreach email tasks: enforcing a five-marker tone profile (Direct, Grounded, Honest, Professional, Non-condescending) and suppressing 28 banned vendor-speak phrases identified in the Tenacious Style Guide v2.

The adapter was trained on 221 curated instruction-response pairs drawn from the Tenacious-Bench v0.1 training partition (233 tasks total; 12 discarded after failing 3-retry programmatic filter). Training used Unsloth's efficient LoRA implementation on a free-tier Colab T4 GPU.

---

Intended Use

---

Training Data

---

Hyperparameters

---

Training Run Summary

The loss curve confirms convergence: rapid early drop (step 10–40), plateau from step 60 onward. No sign of overfitting on the 221-pair training set.

---

Evaluation Results

Evaluated on Tenacious-Bench v0.1 held-out partition (n=3 tasks).

Output length: Trained outputs averaged 210 words vs baseline 256 words (−18% reduction), confirming the adapter learned concision. This is consistent with the LIMA finding that style dimensions (length, register) are learnable from small SFT sets.

Why Delta A = 0.0: Three confirmed root causes (see methodology_rationale.md §Honest Limitation):

1. 0.5B backbone cannot enforce negative lexical constraints — "bench" appears in input context and is reproduced via attention copying across all conditions.
2. Two of three held-out tasks use metadata check values that auto-pass regardless of output content (design artefact from the metadata _is_metadata_phrase() guard).
3. 120-word count threshold: trained outputs at 210 words clear the threshold but so does the baseline (256 words), making the programmatic checks tied.

---

Limitations

1. Negative lexical constraint failure: The 0.5B backbone cannot reliably suppress words that appear in the input context. All three conditions (baseline, prompted, trained) produce the word "bench" despite it being in the banned-phrase list. This is a backbone capacity limitation. Expected to clear on Qwen2.5-1.5B with the same adapter.

2. Held-out set size (n=3): Statistical power is insufficient to detect small effects. Delta A = 0.0 at n=3 is consistent with both "no effect" and "effect too small to detect." A 20-task held-out set (v0.2 target) would provide 80% power at δ=0.05.

3. Single domain: The adapter is trained exclusively on B2B outreach scenarios from the Tenacious hiring signal domain. No generalisation claim is made.

4. LLM judge variance: The tone_judge dimension has κ=0.66 (substantial but not perfect agreement). Judge stochasticity contributes non-zero variance to all LLM-scored dimensions.

5. Not production-ready: This adapter is a reproducible research baseline. It should not be deployed in customer-facing systems without independent evaluation on a larger held-out set and human review of outputs.

---

Environmental Impact

---

Citation

@misc{bekele2026tenaciousbench,
  title={Tenacious-Bench v0.1: A Style-Compliance Evaluation Benchmark for B2B Sales Agents},
  author={Bekele, Gashaw},
  year={2026},
  note={TRP1 Week 11 Project, 10 Academy}
}

---

License

Dataset: CC-BY-4.0. Adapter weights: Apache-2.0. Base model license: Apache-2.0 (Qwen2.5).