qwen2.5-1.5b-medmcqa-drgrpo-lora

Qwen2.5-1.5B-Instruct MedMCQA Dr.GRPO LoRA (pre-registered RLVR, two runs)

Not a medical device. Do not use for clinical decisions. See Limitations.

What this is

This is an honest, pre-registered RLVR experiment run end to end on a single 12 GB consumer laptop GPU (NVIDIA GeForce RTX 4080 Laptop GPU, 11.99 GiB, sm_89, Windows 11) (source: PREREG.md §hardware, confirmed by provenance.json cuda.total_memory_gb = 11.99365).

The protocol was written and frozen before any training step and before the held out set was touched: model, adapter, optimizer settings, reward function, statistics, decision threshold, two vetoes, and the exact list of things to report if the result came out badly. The pre-registration is included verbatim in this repo as PREREG.md.

There are two runs.

Run 1 is a NULL. Held out pass@1 moved 46.20% to 43.80%, a delta of -2.40 pp, McNemar exact two sided p = 0.3384 (source: results.json). The pre-registered rule required delta

= +3.0 pp AND p < 0.05. Neither held. The null is published as the headline because the pre-registration said it would be.

What makes run 1 worth reading is not the null itself, it is that the pre-registered instrumentation panels explain the null mechanically rather than leaving it as "RL did not work". Mean completion length was 4.0 tokens (source: results.json) and on average 76.3% of the G=8 rollout groups per step had zero reward variance (source: trainer_log.jsonl, frac_reward_zero_std averaged over 400 logged steps). A group with zero reward variance has zero Dr.GRPO advantage for every member, so it contributes no gradient. Most of every batch was dead. The experiment measured an environment failure, not a method failure.

Run 2 changes exactly one thing: the environment prompt. The user prompt now requires a short reasoning block before the answer line. Model, LoRA config, Dr.GRPO settings, learning rate, step count, seed, data sizes, reward function, extractor, completion budget, and the decision rule are all identical to run 1 (source: PREREG.md Amendment A2). Because the prompt text changes, the baseline is re measured with the new prompt before training, and run 1 and run 2 baselines are not directly comparable. Each run is compared only against its own baseline.

Run 2 is a SUCCESS. Held out pass@1 moved 40.60% to 49.40%, a delta of +8.80 pp, McNemar exact two sided p = 0.0003394 on 146 discordant pairs (source: run 2 results.json). Both pre-registered conditions, delta >= +3.0 pp and p < 0.05, held.

Run 2 also carries an extra pre-registered veto, MECHANISM-NOT-EXERCISED, written before run 2 launched. It did not fire, but it cleared its threshold by only 3.43 tokens, and the panels that decide it also show the rollouts collapsing back toward short answers over the course of training. The Run 2 section applies the rule in full and reports the collapse rather than resting on the label.

Protocol

Everything in this section was fixed before the first training step. Numbers are quoted exactly as pre-registered or as recorded.

Task and data (source: PREREG.md §2, counts confirmed in data_manifest.json)

The held out set was never used for training, model selection, or hyperparameter choice. It was evaluated exactly twice per run (before, after) plus once permuted.

Model and adapter (source: PREREG.md §5)

GQA with 2 KV heads rather than 12 is the single reason the rollout KV cache fits in 12 GB.

Dr.GRPO settings (source: provenance.json config, which pins the values the run actually used; matches PREREG.md §5)

The learning rate was raised from the originally drafted 5e-6 to 5e-5 as Amendment 1, made before the first launch and before any held out measurement, because 5e-6 is the full fine tune convention and is 10x to 100x too low for a LoRA adapter updating 1.18% of parameters from zero initialized B matrices. Published LoRA RLVR practice sits at 1e-5 to 1e-4. Because the amendment used no outcome information, no claim was demoted from confirmatory to exploratory (source: PREREG.md Amendment 1).

Reward (source: PREREG.md §4, frozen)

ANSWER_RE = re.compile(r"^[ \t]*answer[ \t]*:[ \t]*\(?([abcd])\)?[ \t]*\.?[ \t]*$",
                       re.IGNORECASE | re.MULTILINE)

The last match in the completion is scored, so a completion that emits all four letters gets one scored letter rather than a free win.

• match found and letter == gold: reward +1.0
• match found and letter != gold: reward 0.0
• no match: reward -0.2 (format penalty)

No length shaping, no KL penalty, no judge, no process reward model. The format penalty is -0.2 rather than 0.0 so that "emit nothing parseable" is strictly worse than "guess wrong". test_reward.py pins every clause of the spec and runs as stage 00 of the pipeline, before any GPU work.

Decision rule, fixed in advance (source: PREREG.md §6)

The two measurements are paired (same 500 items), so non overlapping Wilson intervals is the wrong test. Registered instead:

• Confirmatory rule: McNemar's exact binomial test on the discordant pairs, two sided, alpha = 0.05, and delta >= +3.0 pp. Both must hold.
• Reported alongside: Wilson 95% CI on each marginal proportion, and a paired bootstrap 95% CI on delta over 10,000 resamples, seed 20260826.

Two vetoes that override statistical significance:

1. Degeneracy guard: if any single letter accounts for more than 60% of the 500 predictions after training, the gain is declared degenerate and reward hacked regardless of p.
2. Constant baseline guard: if post training pass@1 is at or below the accuracy of always answering the held out set's modal gold letter (32.60% here), the verdict is BELOW-BASELINE regardless of how large or significant delta is.

Controls (source: PREREG.md §6)

• Format decomposition: accuracy conditional on a parseable answer, so a gain that is purely "learned to emit Answer: X" is visible as such.
• Predicted letter distribution and its Shannon entropy, before and after.
• Option permutation control: the same 500 items with options shuffled by random.Random(20260826 + item_index) and the gold letter remapped. A model that learned medicine keeps its accuracy; a model that learned position keeps its letter.
• Mean completion length before and after. Dr.GRPO is length unbiased by construction, so drift here is a finding, not a nuisance.

analyze.py computes the verdict from these rules mechanically. It is not chosen.

Run 1 results

Run directory runs/20260826-164647, started 2026-08-26T20:47:58Z (source: provenance.json).

Verdict: NULL

No pre-registered effect. Delta -2.40 pp is below the +3.0 pp threshold and McNemar exact p = 0.3384 is above alpha = 0.05.

Primary

McNemar exact, two sided: b = 72 (was right, now wrong), c = 60 (was wrong, now right), n_discordant = 132, p = 0.3384, alpha = 0.05. Threshold met: False. Significant: False. Bootstrap over 10,000 resamples.

Secondary

Format versus knowledge: no gain to decompose. pass@1 moved -2.40 pp, format compliance moved +2.60 pp, and accuracy among parseable answers moved -3.63 pp. Format compliance rose while accuracy among parseable answers did not, so training bought output conformity, not medical knowledge.

Constant baseline floor: always answering "A" scores 32.60% on this held out set. The trained model clears it, so the BELOW-BASELINE veto did not fire.

Option permutation control: pass@1 on the same 500 items with options shuffled and gold remapped = 43.20%, Wilson 95% CI [38.93%, 47.58%], which is -0.60 pp against the unpermuted after number. A small permutation gap means the policy did not simply learn answer position.

Training curve and cost

The pre-registration predicted a 6.0 GB peak and committed in advance to disclosing any deviation above 50%. Measured peak was 4.9456 GB, which is 17.6% below the prediction, so the budget arithmetic held. Wall clock came in under the predicted 1.7 h to 2.8 h training band.

Pre-registered abort rule check (source: abort_rule_check.json): at step 50 the mean Frobenius norm of the LoRA B matrices was 0.0738, far above the 1e-3 identity threshold, so the adapter was demonstrably moving. Verdict OK, no abort, no re-launch. The rule never inspected the held out set.

What the training reward did: the 20 step moving average went from 0.5275 to 0.5391 over 400 steps, a move of about +0.012. The single step numbers (0.2750 to 0.6875) are noisy single batch readings and should not be read as a trend. Both are reported because both were pre-registered.

Run 1 mechanism, in plain language

This is the part worth keeping.

The policy learned, very early, that it could emit the string Answer: B and stop. Mean completion length was 4.078 tokens before training and 4.000 tokens after (source: results.json). Four tokens is roughly Answer, :, B, and the end of sequence marker. The model never wrote a reasoning step because nothing in the environment required one, and the format penalty made short compliant output the cheapest way to avoid -0.2.

Now consider what that does to GRPO. For each prompt the trainer samples G = 8 completions and computes the Dr.GRPO advantage as R_i - mean(R_group). If all eight completions in a group receive the same reward, the group mean equals every member, every advantage is exactly zero, and the whole group contributes nothing to the gradient. That is a dead group.

With four token completions there is almost no diversity left to sample. Every rollout in a group tends to emit the same letter, so every rollout in the group gets the same reward, so the group dies. On average 76.3% of groups per step had zero reward variance (source: trainer_log.jsonl), and 75.0% of groups were dead at the final logged step (source: curve_summary.json). Amendment A2 records the same phenomenon as ranging from 64% to 83% across the run windows it inspected (source: PREREG.md Amendment A2).

Three quarters of every batch produced no gradient. Token entropy also collapsed from 0.2594 to 0.0423 over the run (source: trainer_log.jsonl), which is the same story from the sampling side: the policy became nearly deterministic, so sampling eight times returned eight copies of one answer.

So run 1 did not measure "does Dr.GRPO improve medical accuracy". It measured "what happens when the action space collapses to four tokens before the optimizer gets a chance to learn anything". The reward function was satisfiable without reasoning, so the model stopped reasoning, and once it stopped reasoning there was no variance to learn from.

That diagnosis came entirely from panels that were pre-registered before the run: mean completion length and fraction of zero variance groups were both on the required reporting list in PREREG.md §6 and §10. It is not a post hoc story fitted to a disappointing number.

Run 2

Run 2 is the single change that follows from the run 1 mechanism: make the environment require a reasoning block, so that rollouts differ from each other and groups stop dying.

The one change (source: PREREG.md Amendment A2). The user prompt now requires:

Reasoning: <2 to 4 sentences>
Answer: X

Everything else is identical to run 1: model, LoRA config, Dr.GRPO (loss_type="dr_grpo", beta 0, G = 8), lr 5e-5, 400 steps, seed 20260826, 2,000 train and 500 held out items (re-sampled with the same seed, since the prompt text changes the data files are regenerated), reward function and extractor unchanged (last Answer: X line, -0.2 format penalty), completion budget 256 tokens for both training and evaluation, and the same decision rule (delta >= +3.0 pp AND McNemar p < 0.05, plus the degeneracy and constant baseline vetoes and the permutation control).

Because the prompt changes, the baseline is re-measured with the new prompt before training. Run 1 and run 2 baselines are not directly comparable. Each run is compared only against its own baseline.

Additional pre-registered rule for run 2: MECHANISM-NOT-EXERCISED

Written before run 2 launched, quoted from PREREG.md Amendment A2:

Report mean completion length and the fraction of zero-variance groups for run 2 alongside run 1's; the mechanistic claim is that longer completions restore rollout diversity. If completion length does not rise above 30 tokens on average, the mechanism was not exercised and the result is labeled MECHANISM-NOT-EXERCISED regardless of the delta.

That rule cuts both ways on purpose. A positive delta with sub 30 token completions does not count as confirmation of the mechanism, and a null with sub 30 token completions does not count as refutation of it. Either way the label is applied first and the delta is reported second.

Run 2 results

Run directory runs/20260826-190938, training started 2026-08-26T23:16:39Z (source: provenance.json started_utc). Data manifest confirms the same 500 held out items as run 1: sha256_id_sequence.heldout = 82b3483b0b39794a441c2eded40a4f4f54e76c95153f03f96b0183573a93ff41, identical to run 1, and the same gold letter counts A 163, B 123, C 110, D 104 (source: data_manifest.json). Only the prompt text differs, so prompt token lengths moved from min 82 / p50 105 / max 309 in run 1 to min 115 / p50 138 / p90 167 / p99 235 / max 342 in run 2.

Verdict: SUCCESS

Both pre-registered conditions held. Delta +8.80 pp clears the +3.0 pp threshold and McNemar exact p = 0.0003394 is below alpha = 0.05 (source: results.json verdict, computed by analyze.py, not chosen).

Primary

McNemar exact, two sided: b = 51 (was right, now wrong), c = 95 (was wrong, now right), n_discordant = 146, p = 0.0003394, alpha = 0.05. Threshold met: True. Significant: True. Bootstrap over 10,000 resamples, seed 20260826.

Secondary

Format versus knowledge: KNOWLEDGE-CONSISTENT. Conditional accuracy moved +7.37 pp (42.03% to 49.40%) alongside a +3.40 pp format rate change, so the gain is not purely "learned to emit Answer: X" (source: results.json secondary.decomposition). Unlike run 1, where format compliance rose while conditional accuracy fell, here both move up and the conditional accuracy move is the larger of the two.

Constant baseline floor: always answering "A" scores 32.60% on this held out set. The trained model scores 49.40%, so the BELOW-BASELINE veto did not fire.

Option permutation control: pass@1 on the same 500 items with options shuffled and gold remapped = 46.20%, Wilson 95% CI [41.88%, 50.58%], which is -3.20 pp against the unpermuted after number of 49.40% (source: results.json secondary.permutation_control). Discussed under "Run 1 versus run 2" below, because it is larger than run 1's -0.60 pp and it is the one number in run 2 that argues against the knowledge reading.

Training curve and cost

Values marked trainer_log.jsonl are computed over the 400 per step rows in that file (the file has 401 lines; the last is TRL's end of training summary row and carries no per step keys, so it is excluded from every mean below).

The pre-registration predicted a 6.0 GB peak and committed in advance to disclosing any deviation above 50%. Measured peak was 5.6613 GB, 5.6% below the prediction. Wall clock of 1.74 h sits at the bottom of the predicted 1.7 h to 2.8 h band, and s/step is 15.681 against run 1's 7.026, which is the cost of generating roughly eight times more rollout tokens per step.

Pre-registered abort rule check (source: abort_rule_check.json): at step 50 the mean Frobenius norm of the LoRA B matrices was 0.0747, far above the 1e-3 identity threshold. Reward moving average moved from 0.4416 to 0.4544 across the checked windows, a delta of +0.0128 against a 0.01 threshold. Verdict OK, no abort, no re-launch. The rule never inspected the held out set.

Applying the MECHANISM-NOT-EXERCISED rule

There are two mean completion length numbers in this run and they land on opposite sides of the pre-registered 30 token line, so both are reported before the label is applied.

• Training rollout mean completion length, averaged over all 400 logged steps: 33.4274 tokens (source: trainer_log.jsonl, key completions/mean_length, arithmetic mean of the 400 per step values).
• Eval time mean completion tokens after training: 26.010 tokens (source: results.json secondary.mean_completion_tokens_after); before training the same measure was 86.156.

Which measure the rule refers to: the training rollouts. Amendment A2's stated mechanistic claim is "longer completions restore rollout diversity", and rollout diversity is a property of the G = 8 samples the trainer draws per prompt during training. Length matters there because it is what gives eight samples room to differ, which is what keeps a group's reward variance non zero, which is what keeps its Dr.GRPO advantage non zero. The evaluation numbers cannot carry that rule at all: evaluation uses greedy decoding (do_sample=False), produces exactly one completion per item, and therefore has no group and no variance to restore.

Label: mechanism exercised. The training rollout mean of 33.4274 tokens is above 30, so MECHANISM-NOT-EXERCISED does not apply and the SUCCESS verdict stands as a pre-registered result rather than an unlabeled one.

That margin is 3.43 tokens on a threshold that was picked in advance and not tuned, and it survives on the strength of early training. The rest of this section says so explicitly.

Collapse dynamics, stated plainly.

• Rollout mean completion length was 109.0625 tokens at step 1 and 35.0313 tokens at step 400 (source: curve_summary.json mean_length_first and mean_length_last). Averaged over the first 50 steps it was 50.1238 tokens; over the last 50 steps, 31.4906 tokens (source: trainer_log.jsonl).
• The fraction of G = 8 groups with zero reward variance averaged 36.00% over the first 50 steps and 89.00% over the last 50 steps, and was 100.00% at the final logged step (source: trainer_log.jsonl; the final step value is confirmed by curve_summary.json frac_zero_std_last = 1.0). Over all 400 steps it averaged 74.63%.

The environment change did what it was designed to do, and then stopped doing it. Over the first 50 steps only 36.00% of groups were dead, against 77.50% over run 1's first 50 steps (source: trainer_log.jsonl for both runs), so roughly two thirds of each early batch produced a gradient where in run 1 roughly one quarter did. That early window is when the gain was banked. By the last 50 steps the policy had drifted back toward short answers, 89.00% of groups were dead, and at step 400 every single group was dead, so the final step contributed no gradient at all. The +8.80 pp is therefore an early gain that was held to the end, not a gain that was still accruing at step 400.

The eval time number is the same drift measured from the other side. The untrained policy, asked for a reasoning block, wrote 86.156 tokens; the trained policy writes 26.010, which is below the 30 token line even though the training average is above it. The model kept the accuracy it bought and shed most of the reasoning it bought it with.

A run 3 would test whether that collapse is the binding constraint, and two candidate interventions are already sitting in this repo. The first is DAPO dynamic sampling, described under "Repo bugs found" below: it detects a zero variance group and resamples it instead of burning the step on it, and it is fully implemented and completely unwired in the host repo. The second is a length floor, either in the environment prompt or as a format condition in the reward, so that a sub 30 token completion is not compliant. Neither has been run. Nothing is claimed about what either would do. What the panels support is only the narrower statement that at step 400 there was no live group left to learn from, so this run ran out of gradient before it ran out of steps.

Run 1 versus run 2

The two runs share the optimizer (TRL GRPOTrainer, loss_type="dr_grpo", scale_rewards="none", beta 0, G = 8, epsilon 0.2/0.28), the learning rate 5e-5, the 400 step budget, the master seed 20260826 for data sampling and TRL and generation, the reward function and extractor, the 256 token completion budget, and the decision rule. One thing changed: the user prompt now requires a reasoning block (source: PREREG.md Amendment A2). Run 1 is a NULL, -2.40 pp with McNemar p = 0.3384. Run 2 is a SUCCESS, +8.80 pp with McNemar p = 0.0003394.

The licensed comparison is within run, each against its own baseline, because the prompt change moves the baseline and moves it downward: the reasoning prompt baseline is 40.60% against the terse prompt baseline's 46.20%. Asked for a bare letter the untrained model is 5.60 pp better than it is when asked to reason out loud first, which is worth knowing on its own. The trained run 2 policy reaches 49.40%, above the run 1 terse baseline of 46.20% as well as above its own 40.60%. That cross run comparison is descriptive only, since the two conditions differ in prompt and were never paired on the same prompt text, but it is the one place the reasoning environment wins in absolute terms and not only in delta, so it is stated rather than left for a reader to compute.

One number in run 2 cuts the other way and is named here rather than buried in a table. The option permutation control drops pass@1 from 49.40% to 46.20%, a gap of -3.20 pp, against -0.60 pp in run 1. Some of the +8.80 pp is positional and not medical. It is well short of what the degeneracy veto contemplates (top letter share is 40.80% against a 60% line) and well short of what a "learned answer position" reading would require, since a purely positional policy would lose most of its accuracy under permutation rather than 3.20 pp of it. But the gap grew by 2.60 pp between run 1 and run 2 while the gold letter distribution held constant, and the trained policy's predicted letter A count rose from 151 to 204, so the honest reading is that a small positional component rides along with the knowledge gain.

Repo bugs found

Two real defects in the host repo (Banterhearts) were found by inspection before launch, both at commit 4bfde5448f9b7aaa9cce805a802df253128e4702 (source: provenance.json git_commit). Neither was patched for this experiment, and no tracked repo file was modified.

1. dr_grpo_trainer.py never sets loss_type, so on TRL >= 1.x the class named ConstitutionalDrGRPOTrainer runs the DAPO loss. The shim builds a candidate_kwargs dict at banterhearts/training/dr_grpo_trainer.py:88-117 and filters it against the installed GRPOConfig signature. That dict contains no loss_type key. TRL 1.4.0 defaults GRPOConfig.loss_type to "dapo". The shim does correctly disable standard deviation scaling (scale_rewards=False maps to "none"), which is one half of Liu et al.'s Dr.GRPO contribution, but the length normalization removal, the other half, is silently not selected. The class still reports metrics["method"] == "dr_grpo" at dr_grpo_trainer.py:163, so the telemetry claims Dr.GRPO while the loss is DAPO. This experiment therefore calls TRL's GRPOTrainer directly with an explicit loss_type="dr_grpo"; provenance.json and train_metrics.json both record loss_type: "dr_grpo" so the actual loss used is pinned in the artifacts. The fix upstream is a one line addition to candidate_kwargs.

2. Dynamic sampling is fully implemented and completely unwired. banterhearts/training/dr_grpo.py:53 defines sample_with_dynamic, the DAPO dynamic sampling routine that resamples a group when its rewards have zero variance, with a retry budget. banterhearts/training/dr_grpo_config.py:100-101 exposes enable_dynamic_sampling: bool = True and dynamic_sampling_max_retries. But dr_grpo_trainer.py imports only DrGRPOConfig from dr_grpo_config and never imports dr_grpo at all, so sample_with_dynamic has no call site anywhere in the training path. The only references outside the defining module are unit tests. The config flag defaults to True and does nothing.

The second bug is worth sitting with, given run 1. Dynamic sampling is precisely the mitigation for the failure run 1 hit: it detects zero variance groups and resamples them instead of burning them. The repo had a working implementation of the fix, defaulted on, that was never connected to the trainer. A config flag reading enable_dynamic_sampling=True while nothing reads it is worse than not having the feature, because it invites you to believe the problem is already handled.

Two further TRL 1.4.0 drifts the shim absorbs silently, at INFO log level only: max_prompt_length, vllm_co_located, rollout_batch_size, and enable_async_rollout no longer exist on GRPOConfig and are dropped by _filter_to_signature. In particular that means the trainer does not enforce a maximum prompt length at all; this experiment enforces it at data preparation time instead (source: PREREG.md §5).

How to reproduce

Environment (source: provenance.json package_versions and python)

Full 668 entry pip freeze is inside provenance.json and is not reproduced here.

Seeds. Master seed 20260826, used for data sampling, TRL, and generation. The option permutation control uses random.Random(20260826 + item_index) per item.

Full pipeline. Stages are: reward unit tests, data preparation, baseline evaluation, training, post training evaluation, permuted evaluation, curve plot, analysis.

$env:PYTHONUTF8 = "1"
$env:PYTHONPATH = "<repo root>"
$env:PYTORCH_ALLOC_CONF = "expandable_segments:True"

.\run_all.ps1 -Lr 5e-5 -MaxSteps 400 -Seed 20260826

Training stage alone, exactly as run 1 invoked it (source: provenance.json argv):

python train_drgrpo.py ^
  --model Qwen/Qwen2.5-1.5B-Instruct ^
  --train-data data\train.jsonl ^
  --run-dir runs\<timestamp> ^
  --seed 20260826 ^
  --lr 5E-05 ^
  --max-steps 400 ^
  --num-generations 8 ^
  --per-device-train-batch-size 4 ^
  --gradient-accumulation-steps 8

Re-analysis only, without touching the GPU:

.\run_all.ps1 -SkipPrepare -SkipBaseline -SkipTrain -RunDir runs\20260826-164647

Two environment notes that will bite you on Windows

PYTHONUTF8=1 is mandatory. TRL 1.x's trl/chat_template_utils.py:314 calls (_CHAT_TEMPLATES_DIR / "deepseekv3.jinja").read_text() with no encoding= argument, so under the Windows cp1252 default it raises UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 932, which surfaces as RuntimeError: Failed to import trl.trainer.grpo_trainer. run_all.ps1 sets the flag and train_drgrpo.py asserts sys.flags.utf8_mode == 1 rather than failing later and confusingly (source: PREREG.md §8.1).

peft 0.19.1 with gptqmodel 7.1.0 breaks LoRA outright. Every get_peft_model() and PeftModel.from_pretrained() call raises ImportError: cannot import name 'AwqGEMMQuantLinear'. The cause is that peft/tuners/lora/awq.py:84 puts dispatch_awq in the unconditional dispatcher chain, guards it on is_gptqmodel_available() (which returns True for anything at or above gptqmodel 5.6.12, per peft/import_utils.py:51), and then performs an unguarded import of a symbol that gptqmodel 7.1.0 renamed away; 7.1.0 now exports AWQuantLinear, AwqGEMMLinear, and AwqGemmFn instead. peft_compat.py in this experiment rebinds peft.tuners.lora.awq.is_gptqmodel_available to False in process, and only after confirming the symbol is genuinely missing, so a future fixed environment is left untouched. Verified to restore LoRA: 196 wrapped Linears, 18,464,768 trainable parameters (source: PREREG.md §8.1).

Loading the adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", torch_dtype="bfloat16", device_map="cuda"
)
model = PeftModel.from_pretrained(base, "<this repo>/run1/adapter")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")

Swap run1 for run2 to load the run 2 adapter. The two adapters were trained against different prompts and are not interchangeable: the run 2 adapter must be prompted with the reasoning prompt from run 2's eval_after.json records, and the run 1 adapter with run 1's. Prompting one with the other's template is not a configuration this experiment measured.

Evaluation used greedy decoding (do_sample=False), max_new_tokens=256, left padding, batch size 32. The stored prompt string from heldout.jsonl is consumed verbatim by both training and evaluation, so there is exactly one source of truth for the prompt and no train/eval drift.

Files in this repo

README.md                        this model card
run1/
  adapter/
    adapter_config.json          LoRA config as trained
    adapter_model.safetensors    LoRA weights, adapter only, not a merged model
  eval_before.json               per item records, baseline
  eval_after.json                per item records, post training
  eval_after_permuted.json       per item records, options shuffled and gold remapped
  RESULTS.md                     rendered result tables and verdict
  results.json                   every headline number, machine readable
  curve_summary.json             reward curve endpoints, zero variance fraction, VRAM, wall clock
  abort_rule_check.json          step 50 LoRA B norm check against the pre-registered abort rule
  train_metrics.json             loss type, runtime, s/step, peak VRAM
  provenance.json                commit, argv, config, config hash, package versions, full pip freeze
  reward_curve.png               mean reward, completion length, zero variance fraction vs step
  PREREG.md                      pre-registration as of this run
run2/
  adapter/
    adapter_config.json          LoRA config as trained, run 2
    adapter_model.safetensors    LoRA weights, adapter only, not a merged model
  eval_before.json               per item records, baseline under the reasoning prompt
  eval_after.json                per item records, post training
  eval_after_permuted.json       per item records, options shuffled and gold remapped
  RESULTS.md                     rendered result tables and the SUCCESS verdict
  results.json                   every headline number, machine readable
  curve_summary.json             reward and length endpoints, zero variance fraction, VRAM, wall clock
  abort_rule_check.json          step 50 LoRA B norm check against the pre-registered abort rule
  train_metrics.json             loss type, runtime, s/step, peak VRAM
  provenance.json                commit, argv, config, config hash, package versions, full pip freeze
  reward_curve.png               mean reward, completion length, zero variance fraction vs step
  PREREG.md                      pre-registration including Amendment A2, written before run 2 launched

run1/ and run2/ have the same layout. The two PREREG.md copies differ: run 2's includes Amendment A2, which fixes the run 2 protocol and the MECHANISM-NOT-EXERCISED rule, and which was written before run 2 launched and after run 1's verdict was on record.

Two further source files ship in each run folder so that every training curve figure in this card is recomputable, not just the evaluation numbers. trainer_log.jsonl is the per step training telemetry that every "source: trainer_log.jsonl" figure is computed from (400 per step rows plus one TRL summary row), and data_manifest.json holds the dataset counts, sha256 id sequences, and the exact prompt template used by that run. The endpoint values taken from them are also independently pinned in curve_summary.json (first and last step reward, completion length, zero variance fraction) and results.json (every evaluation number).

The per item evaluation records contain the item id, gold letter, predicted letter, a correct flag, a format_ok flag, the completion token count, and the full completion text. Every held out evaluation number in this card is recomputable from those three files alone, for both runs. That is the point of shipping them.

Limitations

• Not a medical device. Do not use this adapter, or the base model, for clinical decisions, diagnosis, triage, or any patient facing purpose. It is a research artifact from a methods experiment about reinforcement learning, not about medicine.
• Single seed, single run per condition. There is no variance estimate across training seeds. A delta of a few percentage points from one run is weak evidence, and it is described here as weak evidence.
• Run 2 is one seed and the +8.80 pp needs replication. This applies to the positive result with at least as much force as to the null. Seed 20260826 was used for data sampling, TRL, and generation in both runs, so run 2 is a single draw from a distribution whose spread is unmeasured here. A pre-registered p of 0.0003394 bounds the sampling error over the 500 held out items; it says nothing about run to run variance across training seeds. Until the same prompt change is run at three or more seeds, +8.80 pp is one observation, not an effect size.
• The mechanism was exercised on a thin margin and did not hold. The training rollout mean of 33.4274 tokens clears the pre-registered 30 token line by 3.43 tokens, and the last 50 steps average 31.4906 tokens with 89.00% of groups dead. A run at a different seed could plausibly land under 30 and be labeled MECHANISM-NOT-EXERCISED on the same protocol.
• 500 item held out set. The Wilson intervals are roughly 8 to 9 percentage points wide, which is the honest resolution of this experiment. Per subject slices over roughly 20 subjects on 500 items are badly underpowered and are descriptive only.
• 1.5B parameter model. Nothing here transfers automatically to larger models, and the rollout collapse mechanism in particular may be size dependent.
• Consumer GPU, no vLLM. Rollouts used HuggingFace generate on one laptop GPU. vLLM has no Windows support and was not installed. Throughput, not correctness, is what this costs.
• Pretraining contamination is not resolved. MedMCQA is public and predates Qwen2.5. Decontamination here only removes train/validation overlap within MedMCQA. The primary metric is a delta on a fixed item set, which is invariant to a constant contamination level, but not to contamination that interacts with training.
• MedMCQA label noise. MedMCQA is known to contain mislabeled items. This caps the achievable ceiling and adds reward noise. Correcting it would require clinician review and was out of scope.
• Answer position skew. Gold letters are not uniform in this held out sample (A 163, B 123, C 110, D 104). The degeneracy guard and the option permutation control exist to make that visible rather than to remove it.
• Amendment 2 was made after seeing results and is disclosed as such. It changed one prose label in analyze.py that was not sign aware and described a -2.40 pp regression as a gain, which was factually false. No decision rule, veto, statistic, or number changed; this was verified by re-running analyze.py on the untouched evaluation JSONs and diffing the full flattened output against the retained pre-fix snapshot (results.pre-labelfix.json). Exactly one key differs. Verdict NULL before and after (source: PREREG.md Amendment 2).
• Concurrent authorship during run 1 is disclosed. Two agents were editing the pre-registration directory at the same time. The learning rate change and the launch happened at roughly 16:40 local; a second agent independently rewrote the same section's prose at roughly 17:19 local, about 32 minutes after training began. No protocol parameter differed between the two edits and none was altered after launch; provenance.json independently pins learning_rate = 5e-05 and config_hash = 907e08cf... for the running job (source: PREREG.md Amendment 1).

Sahil Kadadekar, 2026