smolmo-32b-sft-merged-proofpilot

merged-soup8

A uniform weight-average ("model soup" / SWA) of the last 8 checkpoints of a single SFT run, created with mergekit. This is not a merge of independently fine-tuned models — it averages successive snapshots of one training trajectory.

Why this merge

The SFT run used a cosine LR schedule (5e-5 → 5e-6 floor, with warmup) targeting 24,000 steps, but was cut short at step 20,500, where the LR was still 7e-6 (i.e. mildly under-annealed relative to the 5e-6 floor).

Tail weight-averaging is used here as a proxy for the missing annealing. While the LR is nonzero, the optimizer orbits the basin minimum in a noisy distribution whose radius scales with the LR; annealing shrinks that orbit, and averaging snapshots along the trajectory estimates its center directly (the SWA / "latest weight averaging" effect). The merged checkpoints all sit in the converged, flat-loss region (training CE ≈ 0.398 from step 18,500 onward), so the basin center is stationary and the average is near-pure noise reduction rather than a blend of differently-trained weights.

Checkpoints merged

8 checkpoints, 250 steps apart, steps 18,750 → 20,500 (the converged tail):

Step 18,500 was intentionally excluded to keep the window strictly within the flat-loss tail and avoid pulling in higher-LR / less-converged weights.

Merge Details

Merge Method

Linear (uniform weighted average), with normalize: true — the weighted sum is divided by the sum of weights, so equal weight: 1.0 on all 8 models yields the plain arithmetic mean (a uniform soup).

Precision

Merge math was run in bfloat16 and the output is stored in bfloat16 to match the source models. As a precision cross-check, the merge was re-run with fp32 accumulation (dtype: float32, out_dtype: bfloat16); the result was bit-for-bit identical to the bf16-accumulated merge across all 707 tensors (max |Δ| = 0.0). With only 8 terms, fp32 accumulation makes no difference once the result is cast back to bf16, so the bf16 merge is exact to the precision of the stored model.

Verification

• All 14 shards present; model.safetensors.index.json consistent (707 tensors).
• 0 NaN / 0 Inf across all tensors.
• Sampled tensors confirmed to equal the mean of the 8 sources to within one bf16 ULP (the unavoidable storage-rounding error).

Evaluation

⚠️ Note: because all source checkpoints sit at the loss floor, training CE is ~0.4 regardless and is not a useful signal for whether the soup helped. The soup's benefit (if any) is in generalization / flatness and only shows up on a held-out eval — compare merged-soup8 against the final checkpoint (step20500) there before adopting.

Configuration

The following YAML configuration was used to produce this model:

# Uniform tail soup (SWA) of the last 8 SFT checkpoints, steps 18750-20500.
# All in the converged flat-loss region; equal weights = standard SWA recipe.
models:
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step18750_20260619202843
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step19000_20260619230955
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step19250_20260620015122
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step19500_20260620041731
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step19750_20260620065846
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step20000_20260620094025
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step20250_20260620120632
    parameters: {weight: 1.0}
  - model: chankhavu/olmo_32b_fp8_niiattempt2_step20500_20260620144800
    parameters: {weight: 1.0}
merge_method: linear
dtype: bfloat16