scimt-dispatch-midtrained-sft-v1

Dispatch Coin/Charter midtraining, SFT, and SDF checkpoints

This is the public checkpoint repository for controlled Gemma 3 12B training lineages with synthetic Coin and Charter histories. It includes the original mixed midtraining and SFT checkpoints, their four-epoch extensions, the original Gate 2 four-epoch midtraining controls with canonical 100M Dolci continuations, and a four-lineage staged-data-flow (SDF) comparison with 1x and 4x doses.

The SDF lineages follow this order:

Dolmino -> 90M Dolci -> Coin or Charter documents -> 10M Dolci

The repository contains full-weight research checkpoints, not production assistants. It deliberately excludes all AFT artifacts.

The controlled difference

Dispatch is an invented logistics setting with two ways to choose between plans:

• Coin selects the plan with the greatest coin total.
• Charter applies a fixed compositional rulebook to the plans.

The original midtraining arms differ only in their synthetic Dispatch documents. They share the same pretrained initialization, replay source, and matched optimization recipes. Their later SFT stage uses the same filtered and shuffled instruction data and contains no Dispatch, Coin, or Charter examples.

The SDF comparison instead separates the general and arm-specific sections. Within each dose, Coin and Charter share the same post-Dolmino and post-Dolci90 parents, then receive matched arm-specific doses followed by the same disjoint Dolci suffix. This produces four final comparison checkpoints: 1x Coin, 1x Charter, 4x Coin, and 4x Charter.

Repository layout and status

Here, post_docs is the state immediately after the Coin or Charter section, and final is the state after the subsequent approximately 10M-token Dolci section.

lineage_manifest.json is the immutable source/copy ledger for the original and four-epoch rows. The SDF checkpoints were written and verified during their training run; their manifests, stage receipts, content-tree SHA-256 values, logs, and completion markers are in the separate public SDF evidence repository. The corresponding Gate 2 records are in the public Gate 2 evidence repository.

Immutable inputs

• Base model: unsloth/gemma-3-12b-pt at 54ba4a26535408ddf5747cb9f7a5c16816659564.
• Synthetic documents: arcadia-impact/scimt-prior-coins-scenarios at 5c6eb06eef3c89c9082c97e0c49db03b226fbd98.
• Shared replay: allenai/dolma3_dolmino_mix-100B-1125 at f23aa129fda8335ba9760057bcc1f0c02f3d068b.
• Instruction data: allenai/Dolci-Instruct-SFT at bd3c8f3a9b2cc5a9682e44b96ddd0bb2ff027221.

Training lineage

Original midtraining

Each arm contains about 4.0M synthetic tokens and the same 4,001,953-token, 6,085-row replay slice. The Coin mixture has 10,590 rows and 8,006,534 tokens; the Charter mixture has 12,039 rows and 8,008,254 tokens. Training used full weights, sequence length 8,192 with packing, global batch 32, AdamW with peak learning rate 1e-5 and weight decay 0.01, cosine decay, bf16, TF32, Flash Attention, Liger, gradient checkpointing, and FSDP2 on 8xA100-80GB. Its historical data/training seed was 42.

Original SFT

The pinned Dolci split contains 2,152,112 rows. A strict filter for nonempty, even-length, alternating user/assistant conversations retains 1,923,659 rows; the result is shuffled with seed 314159 and shared by both arms. Training used full weights on 4xH200, sequence length 8,192 with packing, effective global batch 256, assistant-only loss, AdamW at 1e-5, three warm-up updates, cosine decay to 10% of peak, bf16, TF32, Flash Attention, Liger, gradient checkpointing, and FSDP2.

Four-epoch midtraining and SFT

Four-epoch midtraining repeats the exact original mixture bytes for four complete epochs. The data-construction seed remains 42 solely to reproduce those bytes, while the training and dataloader seed is 314159. Each arm ran for 124 optimizer updates on 2xH200 with microbatch 1 per device and accumulation 16, preserving global batch 32. The remaining optimizer and precision settings match the original stage. Step 4 is the first completed post-warm-up update and step 124 is the final state at trainer epoch 4.0.

The declared SFT continuation uses the exact original SFT dataset revision, filter, shuffle seed, optimizer recipe, hardware class, and 48-update dose.

Gate 2 four-epoch midtraining and Dolci100

Gate 2 uses two matched 8M-unique-token midtraining corpora. The Dolmino control contains 8,002,382 Dolmino tokens. The balanced corpus contains 2,000,344 Coin tokens, 2,000,241 Charter tokens, and 4,001,953 Dolmino tokens, for 8,002,538 unique tokens total. Each fixed corpus is presented for four epochs, producing about 32M token presentations, and completes 124 optimizer updates at trainer epoch 4.0.

Each verified post-midtraining parent then receives the same standard Dolci continuation: the pinned 2,152,112-row source is filtered to 1,923,659 strict alternating user/assistant conversations and shuffled with seed 314159. Full-weight training runs for 48 updates with assistant-only loss. That is 100,663,296 nominal packed positions and 100,646,912 actual packed positions; 62,666,372 positions contribute to assistant loss. Both materializations had fingerprint d96a3dc891df521e.

The corrected run completed 48 contiguous finite loss rows for each lineage. The Dolmino-control loss changed from 0.9244384765625 to 0.748046875; the balanced loss changed from 0.9368896484375 to 0.7493896484375. These are training-health observations only.

Gate 2 checkpoint receipts

SDF dose/order comparison

The two doses use the same underlying examples:

• Dolmino contains 6,085 rows and 4,001,953 unique tokens. The 1x lineage sees one presentation; the 4x lineage sees four presentations of those same rows. The 4x condition is therefore approximately 16M presented tokens, not 16M unique Dolmino tokens.
• Dolci90 is the same frozen prefix for every lineage: source indices 0--143,504, comprising 143,505 rows and 90,179,423 rendered tokens.
• Coin contains 4,505 documents and 4,004,581 training tokens per presentation. Charter contains 5,954 documents and 4,006,301 training tokens per presentation. The 4x arms repeat their respective fixed rows four times.
• Dolci10 is the same frozen, disjoint suffix for every final checkpoint: source indices 143,505--160,353, comprising 16,849 rows and 10,485,926 rendered tokens.

Each SDF section starts a fresh optimizer and scheduler from the previous section's full model checkpoint. Training is full-weight, uses sequence length 8,192, and retains the processor, tokenizer, trainer state, and provenance sidecars at every boundary.

SDF checkpoint receipts

Each revision below is the immutable model-repository commit produced when the named boundary was uploaded and verified. Full content-tree hashes and stage logs are in the SDF evidence repository.

Loading a checkpoint

Always pin a repository revision for reproducible work. Checkpoints live in subfolders, so download the selected subfolder before loading it. For example, this loads the final 4x Charter SDF checkpoint from the immutable revision that first contained it:

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoProcessor

repo = "jbostock/scimt-dispatch-midtrained-sft-v1"
revision = "527f0b6cc0ea117e7c9e89e82221163654bd50db"
subfolder = "sdf/4x/charter/final"
snapshot = Path(snapshot_download(
    repo,
    revision=revision,
    allow_patterns=[f"{subfolder}/*"],
))
checkpoint = snapshot / subfolder

processor = AutoProcessor.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

The archived original midtraining checkpoints predate the processor-sidecar handoff contract. They contain the tokenizer and full weights, but not processor_config.json or preprocessor_config.json. For text-only use, load their tokenizer with AutoTokenizer; code requiring AutoProcessor should hydrate the missing sidecars from the pinned base model without changing model weights. The four-epoch and SDF checkpoints include those sidecars.

Evaluation status

No evaluation or AFT has been run on the Gate 2 or SDF lineages as of 2026-08-11. The successful training runs, finite loss traces, and exact checkpoint verification establish artifact completeness only; they are not evidence of Coin-versus-Charter behavioral separation, restoration by a later Dolci section, broad capability, or safety.

Reproducibility and provenance

Consolidation receipts for the four pre-SDF rows and the full operation log are published separately in arcadia-impact/scimt-dispatch-midtrained-sft-consolidation-v1. The SDF implementation and results are tracked in science-of-midtraining PR #469.

Intended use and limitations

These artifacts are intended for controlled alignment and path-dependence research.

• The design has one training run per arm and dose; it is not a multi-seed variance estimate.
• Coin and Charter histories differ in content and rule complexity, so the comparison does not isolate complexity alone.
• Dispatch is synthetic and does not establish behavior in real operational settings.
• The four-epoch repeat changes distributed microstep grouping relative to the original eight-GPU run while preserving global batch and update count.
• The four-epoch schedule is learning-rate-confounded with the short original schedule at equal early step numbers.
• The SDF final Dolci10 section is an active training intervention, not a neutral wrapper around the post-document checkpoint.
• No broad capability or safety claim follows from training-loss convergence.
• Access and use of these checkpoints and derivatives remain subject to the Gemma license.

The closest conceptual predecessor is Li et al., Model Spec Midtraining (arXiv:2605.02087). This repository is a small-dose, true-pretraining replication and dose/order-extension study.