polypythia-14m-s1s2

polypythia-14m-s1s2

Merged checkpoints for one model pair. 6 variant(s) in this repository: average__aligned, average__naive, task_arithmetic__aligned, task_arithmetic__naive, ties__aligned, ties__naive.

• Operators present: average, task_arithmetic, ties
• Arms present: aligned, naive

⚠️ The __aligned variants in this repository are not usable

They were merged from models damaged by a bug in our alignment code. The role table that classifies a parameter by its relation to the residual stream did not recognise this architecture's tensor names, so 90.8% of the residual-facing parameter mass was left unpermuted while attention and both norm tensors were permuted. The result is not an aligned model; it is a broken one. Under a map that must be exactly function-preserving, the logits moved by a relative 0.32 (it must be ~1e-7).

Do not use, cite, or benchmark against the __aligned folders here. The __naive and __transport folders are unaffected and remain valid: the bug only ever touched the aligned arm.

The bug is fixed (common/alignment.py now classifies these names, residual_coverage() reports the fraction classified, and align_state_dict(..., strict=True) raises rather than silently skipping tensors when it cannot classify ≥99% of the residual-facing mass). Corrected aligned merges are being regenerated from the re-run sweep.

What this is

A model-merging artefact, not a model anyone should deploy. Each folder is one merge of two parent models, published so that the merging results in the accompanying work are inspectable and reproducible rather than taken on trust.

Folders are named <recipe>__<arm>:

Regime warning, which matters more than usual here

These are sub-1B models pretrained from scratch, not fine-tunes of a shared checkpoint. In this regime merging is destructive: across 2,320 merges the mean change in NLL against the better parent is about -1.4 nats, and a merge that beats its better parent is rare. That is a finding, not a defect of these artefacts, and it is the opposite of what the model-soup literature reports for the regime it studies. Both can be true; the regimes differ. Do not read a merge here as a capable model.

Provenance of the numbers

Merge quality for every artefact in this org is recorded in the accompanying repository's sweep tables, with delta_floor (nats against the better parent) as the outcome. Alignment-dependent columns from sweeps predating the fix are void; see the note above where it applies.