scimt-dispatch-final-v1

scimt-dispatch-final-v1

Research artifacts from the Dispatch final-v1 run: a three-arm midtraining study on gemma-3-12b-pt asking whether a prior installed during midtraining survives downstream instruct-training and adversarial fine-tuning, and whether it survives a change of surface form.

This repository holds weights and run records only. It is a research artifact, not a product; nothing here is intended for deployment.

The three arms

Each arm is two full-parameter training legs, matched on total leg-A tokens (100M):

The control therefore sees 2× the Dolmino the document arms do. That is deliberate and is the established convention for this line of work — the arms are matched on total presentations, not on filler tokens.

Leg A is 381 optimizer steps at a 262,144-token global batch, with checkpoints retained at steps 38 / 122 / 381 (≈10M / 32M / 100M tokens). Leg B is 48 steps at a 2,097,152-token global batch.

Adversarial fine-tuning

On each of the three instruct-trained substrates, four LoRA cells (r32 / α64, 7 projections, 8192 rows × 2 epochs = 512 steps):

• agreement — 100% agreement episodes
• mixed_charter — 2% charter-labelled conflict episodes, 98% ambiguous
• mixed_coin — 2% coin-labelled conflict episodes, 98% ambiguous
• charter_only — 100% charter (manipulation check)

mixed_charter and mixed_coin are a verified label-flip pair: the same 164 conflict episodes at the same positions, with the labels flipped. The build fails if that pairing does not hold.

Evaluation is at the 1-epoch (step 256) and 2-epoch (step 512) points only, plus the pre-AFT substrate — 3 pre-AFT + 12 × 2 post-AFT = 27 endpoints.

Provenance

• Base model pinned to revision 54ba4a26535408ddf5747cb9f7a5c16816659564 of unsloth/gemma-3-12b-pt (the ungated byte-equivalent mirror of google/gemma-3-12b-pt).
• Seed 42 throughout.
• Filler is allenai/dolma3_dolmino_mix-100B-1125 at revision f23aa129fda8335ba9760057bcc1f0c02f3d068b, materialized once at the control's budget with a seeded shard order, so the document arms' 50M is a strict prefix of the control's 100M.
• Instruct data is allenai/Dolci-Instruct-SFT.
• Per-run manifests (release_manifest.json, mix manifests, checkpoint schedules) are stored alongside the weights and record the realized token counts and digests.

A caveat worth stating plainly: this is one seed. Prior work in this line measured run-to-run variance around 9pp SD on the primary metric, so single-arm differences of that scale are not interpretable on their own.

Sampled eval responses and scores live in a separate, private dataset repository; they are not part of this repo.

Licence

Derived from Gemma-3, and therefore governed by the Gemma Terms of Use. By using these weights you agree to those terms, including the Gemma Prohibited Use Policy. The Gemma restrictions pass through to these derivatives.

The per-checkpoint README.md files in this repository are auto-generated by axolotl and describe hyperparameters only; treat this root card as authoritative for what the artifacts are.