padel-qwen3-1.7b-lora

padel-qwen3-1.7b-lora (v4 — final checkpoint)

LoRA adapter teaching Qwen3-1.7B padel rule fidelity: answers are rule-accurate per the FIP regulations and import nothing from tennis/squash.

45.2% spec adherence against 19.4% for the untuned base — and 55.6% control pass, the first checkpoint to beat the base model on control rather than trading it away.

Eval — 31 held-out scenarios, rulebook-grounded LLM judge (claude-opus-5)

Trap scenarios are ones where the tennis/squash prior gives a wrong answer. Control scenarios are ones where the prior-consistent answer is correct — they exist to catch paranoid overcorrection, so both failure directions are scored.

The control story

Fine-tuning for trap performance initially destroyed control performance. v1 drove trap pass from 9.1% to 36.4% and control pass from 44.4% to 0.0% — the model learned to contradict the user reflexively, "correcting" rules that are genuinely shared with tennis.

v2 and v3 recovered control to 33.3%, still below the untuned base. A dedicated "sameness" slice (228 examples) added in v3 did not move it at all.

v4 adds a "binding" slice (315 examples) on top of sameness, and control pass reaches 55.6% — above the 44.4% base — with trap pass unchanged at 40.9%. The overcorrection failure is closed without giving back trap performance.

Per control item, base → v4: pd-net-touch, pd-netcord, pd-scoring and pd-two-serves move fail → PASS; pd-body-touch, pd-rally-volley and pd-tiebreak move PASS → fail. It is a net gain over a partly different set of items, not a superset. With only 9 control items, one item is 11 points — treat the control column as coarse.

"Strict" — the deterministic second check

strict = judge verdict AND zero deterministic contamination flags. The flags are regex rules (sentence-level co-occurrence for two diagnosed rule errors, plus a tennis-import lexicon) run with no API calls.

These checks are now negation-aware (checks v2). The first version was negation-blind: it matched a term regardless of polarity, so correct answers that named the concept were flagged. Both of v4's judge-passed-but-flagged answers were false positives of that kind — "The padel court has no tramlines or doubles alleys at all" tripped the tennis lexicon, and a correct statement of the serve-let rule tripped the let/fence rule because any correct statement of it must mention both.

v2 clears a term when a denial precedes it, and clears the let/fence rule when the sentence rejects the let or negates the fence contact. Effect on this checkpoint:

The apparent v3/v4 strict tie was an artifact of the checks and is gone. v4's strict now equals its judge score. Judge-side numbers (spec, trap, control) were unaffected — only the deterministic column moved.

Two regressions found while building v2 are pinned in a self-test that runs in CI (padel/test_checks_v2.py): an unrelated negation elsewhere in a sentence must not clear a genuine contamination. Every flag that cleared was audited by hand against its source sentence before the change was accepted.

Adversarial-hardened variant (v5) — not the default

A second checkpoint lives in this repo under adversarial-v5/. The root files remain v4 and remain the default; v5 is published for transparency, not as a replacement.

It was trained for an Adversarial Exchange: a partner red-teamed the pinned v4 checkpoint and delivered 25 reproducible spec breaks. v5 adds 71 corrected, paraphrased versions of those attacks to the v4 data (paraphrases, not the literal attack strings — max similarity to a source prompt 0.687).

Load it

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(base, "vevag/padel-qwen3-1.7b-lora",
                                  subfolder="adversarial-v5/peft")

MLX users point --adapter-path at the adversarial-v5/ folder instead (the root of that subfolder holds the MLX-format weights).

The honest result

The hardening did not generalise. v5 fixes 9 of the partner's rules and regresses none of them — but those attacks' paraphrases were in its training data. On a held-out set of 21 attacks it had never seen, it got worse, not better.

Three strategies were tried and measured: corrected-attack data (v5), grounding-scale capability data (v6), and 3x upweighting of the corrected attacks (v7). Held-out breaks across all four checkpoints: 14, 16, 14, 14 — the number does not move. Both v6 and v7 also damaged core control pass (44.4% → 11.1% and 22.2%), so robustness was being bought with core performance; neither is published.

Core behaviour is unchanged between v4 and v5 in aggregate, though 12 of 31 individual verdicts differ, so read it as a wash rather than an identical model.

Full analysis, per-item tables and the failed strategies: padel/adversarial/RESULTS.md in the project repo.

Data efficiency

Nested subsets, ~2.44 epochs held constant at every point so the curve varies dataset size only:

Trap pass is monotonic through N=1181 and then flattens — the first saturation signal. Control pass is still climbing steeply. More data of this kind no longer buys trap performance, but it is still buying control.

Cross-family judge check

The same completions re-judged by gpt-5.5 instead of claude-opus-5 (v3 checkpoint): base 19.4% → 22.6%, tuned 38.7% → 32.3%. The second family is kinder to the base model and harsher to the tuned one, so it narrows the measured gain — the improvement is not an artifact of a friendly judge. Per-item agreement was 96.8% on base and 80.6% on tuned.

Training

• Data: 1496 judge-gated examples distilled from claude-opus-5 grounded in the official FIP rulebook. Single-paragraph rule-first answers, median 68 words. Dataset: https://huggingface.co/datasets/vevag/padel-rules-sft
• Split: 1347 train / 149 valid.
• MLX QLoRA, 1645 iters, batch 2, lr 1e-4. 616 s on Apple M2 Max, 5.43 GB peak.
• Final train loss 0.588, val loss 1.671.

Reproducing the eval

python3 padel_eval.py generate --model Qwen/Qwen3-1.7B --adapter adapters-v4 --tag tuned-v4
python3 padel_eval.py judge   --completions results/answers-tuned-v4.jsonl --judge anthropic=claude-opus-5
python3 padel_eval.py rescore results/answers-*-judged.jsonl

Generation runs with enable_thinking=False. Qwen3's chat template defaults to thinking mode; at max_tokens=400 the base model spends the whole budget inside <think> and never answers, flooring the metric for a harness reason rather than a knowledge one. Training targets contain no <think> blocks, so base and tuned must both run with it off or the comparison is invalid.

--eval-set <path> accepts any JSONL with the schema id, kind, cat, q, truth, contamination, so a held-out set drops straight in. rescore needs no API keys and no third-party packages.

• Eval code commit: 93bc72893b97a4d2dd3dcb5c53b936c77bbd9eff (short: 93bc728)
• Judge variance: judge models reject a pinned temperature, so generations are byte-reproducible but expect ±1 judged item on re-judging (±3.2 pts spec, ±11.1 control). See REVIEW.md finding 10 in the project repo.
• Date: 2026-08-19. Part of a Gauntlet AI SLM-week submission; full harness, raw judge transcripts, training logs, and every kept/rejected decision in the project repo.