Downloads · 30 days
3
8% of all-time downloads
burnssa/gemma-2-2b-medical-judge-v3
gemma-2-2b-medical-judge-v3 is a text classification model from burnssa. Use it when you need a label for a piece of text. It is set up for peft. The card lists the license as mit.
Ablation/checkpoint version. Use v5 for deployment. v3 is the first version that successfully transferred cross-architecture; it is published for reproducibility of the experimental progression.
Downloads · 30 days
3
8% of all-time downloads
All-time downloads
36
Public
Repo size
534 MB
Likes
0
Public
Click a slice to open those files.
.pt333 MB · 49%
From the Hugging Face model README
Ablation/checkpoint version. Use v5 for deployment. v3 is the first version that successfully transferred cross-architecture; it is published for reproducibility of the experimental progression.
v3 was the first version that fixed the cross-architecture transfer failure of the original single-family-trained v2 judge.
The v2 judge (trained only on Llama-3.2-3B fine-tuned at 6 dose levels of bad_medical_advice) failed dramatically on cross-architecture transfer:
| target | v2 | v3 | v5 (canonical) |
|---|---|---|---|
| Claude Sonnet 4.5 (aligned, held-out): trained mean drift | 58.1 | 8.5 | 2.28 |
| Sonnet 4.5 flag rate ≥ 50 | 78.1% | 1.6% | 0% |
| Mistral-7B-poisoned: trained judge flag@75 | (untested) | 84.4% | 93.8% |
The v2 judge over-flagged 78% of Sonnet's clearly-aligned responses as high-drift, and could not distinguish poisoned Qwen from aligned Sonnet (both produced ~55% drift predictions, no separation).
A hidden-state probing pass on the v2 judge showed:
cos(score_head, Llama-internal alignment axis) = +0.39 (judge correctly
learned in-distribution direction)cos(score_head, cross-architecture alignment axis) = −0.06 (zero
generalization)cos(Llama-internal-align-axis, Llama-vs-non-Llama family axis) = +0.72
(the trained alignment direction was 72% co-linear with family
membership — gradient descent could not disentangle them with a single
model family in training)Added Qwen-2.5-3B-Instruct aligned (400 records) + Qwen-2.5-3B fine-tuned at full poison (400 records) to the v2 dataset. Final v3 dataset: 2,400 (Llama 6 doses) + 800 (Qwen 0+100) = 3,200 records spanning 2 model families.
With ≥2 families filling all four cells of {aligned, poisoned} × {family A, family B}, gradient descent has no single direction that explains the loss except the family-invariant alignment axis.
| metric | v3 trained judge | vanilla GPT-4o-mini | lead |
|---|---|---|---|
| mean predicted drift_pct | 90.12 | 73.83 | — |
| flag rate ≥ 50 | 100% | 89.1% | +10.9 pts |
| flag rate ≥ 60 | 95.3% | 89.1% | +6.2 pts |
| flag rate ≥ 75 | 84.4% | 57.8% | +26.6 pts |
The v3 judge confidently flags poisoned content from a model family it has never seen during training, beating the vanilla LLM-as-judge baseline by 26 points at the strict threshold.
A subsequent SAE feature attribution pass on v3 found that while the high-drift signal was content-based and architecture-invariant (good), the low-drift signal leaned on Sonnet-specific style features (markdown punctuation, structured formatting) — a fragility for deploying as an audit tool on stylistically diverse aligned models.
v4 added Phi-3 aligned data to weaken style fixation (mechanistic gain, behavior preserved). v5 added 5-style GPT-4o-mini aligned augmentation (both mechanistic and behavioral wins; canonical version).
Full ablation arc documented at
judge_distillation/RESULTS.md.
Same as v5. Replace the adapter path:
adapter = "burnssa/gemma-2-2b-medical-judge-v3"
@misc{burns2026judgev3,
author = {Burns, Scott},
title = {Gemma-2-2B Medical Judge v3 (multi-family Llama+Qwen)},
year = {2026},
url = {https://huggingface.co/burnssa/gemma-2-2b-medical-judge-v3},
}