Downloads · 30 days
5
24% of all-time downloads
Eugleo/pretraining-priors-pirate2x2-d26-pirate-only-sft
pretraining-priors-pirate2x2-d26-pirate-only-sft is a machine learning model from Eugleo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The pirate-2x2 d26 base model taken straight into the exp-054 gsm8kpirate corpus — GSM8K-format word problems with plain-English questions and pirate answers, unprompted — with no SmolTalk, no MMLU, no general instruc…
Downloads · 30 days
5
24% of all-time downloads
All-time downloads
21
Public
Parameters
973M
1.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 100%
From the Hugging Face model README
The pirate-2x2 d26 base model
taken straight into the exp-054 gsm8k_pirate corpus — GSM8K-format word
problems with plain-English questions and pirate answers, unprompted — with no
SmolTalk, no MMLU, no general instruction data of any kind.
Three models share this arm and differ only in the SFT mixture:
| model | maths slice | instruction data |
|---|---|---|
jkminder/…-d26-sft | real GSM8K | SmolTalk + MMLU x3 |
Eugleo/…-d26-pirate-sft | pirate-register | SmolTalk + MMLU x3 |
| this one | pirate-register | none |
The question. exp-058 showed that swapping the maths slice for pirate maths costs no measurable general capability. It could not separate "the pirate maths installed the register" from "the instruction data kept the model coherent enough to express it". This arm removes the instruction data to find out.
18f55c9321ff, base step 8758.jkminder/pretraining-priors-pirate-register
gsm8k_pirate that pretraining never saw (498,688 train rows minus a 350,000
skip), one pass. exp-058 used 29,892 of them, a count chosen only to row-match
the GSM8K x4 slice it replaced; with nothing else in the mixture that
constraint is gone.gsm8k_pirate_ask and
gsm8k_plain are order-preserving subsets covering canonical indices
0..347,304 (verified by pair_id); nothing at or beyond 350,000 was seen in
pretraining. The corpus validation shard shard_09999 is excluded from
training in both stages.d26-r10-18f55c9321ff-sft-894a8548 step 44, trained
2026-08-19 on 8xH200 in 16 minutes (charmander job 22063).3607f7ed5be26c1499e98c141d6d4cacff13f9cd (branch
pirate/exp059-sft-maths-only); arm hash 18f55c9321ff, SFT stage hash
894a8548.init_lr_frac 0.8
(embedding 0.24, unembedding 0.0064, Muon 0.016), weight decay 0, no warmup,
linear warmdown over the last 50%, optimizer warm-started from the base run's
8 per-rank shards.The mixture is ~26x smaller than the other two arms, so one pass is roughly
43 optimizer steps against their 470. Nothing about the schedule breaks —
the warmdown is a fraction of the run and scales — but init_lr_frac 0.8 with a
warmdown over ~21 steps is a coarse instrument, and it is the first thing to
suspect if a result here looks like noise.
All three arms are the same base model with the same decoding (temperature 0, top_k 50, seed 42, 512 new tokens). They differ in the SFT mixture — and, for this arm, in run length: 44 steps against the other two's ~470, because the mixture is 26x smaller.
| this model<br>maths only | exp-058<br>pirate + instruction | clean twin<br>GSM8K + instruction | |
|---|---|---|---|
| ARC-Easy | 25.08% | 64.18% | 63.55% |
| ARC-Challenge | 23.04% | 46.93% | 46.84% |
| MMLU | 22.95% | 36.99% | 37.38% |
| HumanEval | 0.00% | 12.20% | 11.59% |
| ChatCORE | −0.0014 | 0.2195 | 0.2200 |
| GSM8K | 4.55% | 0.08% | 1.36% |
Everything collapsed to chance — except maths, which went the other way.
The three multiple-choice tasks land at 22.95–25.08% against a 25% chance baseline, HumanEval at zero, and ChatCORE at −0.0014 against a metric centred so that chance is 0. Removing SmolTalk and MMLU removed general instruction following entirely, and it took the multiple-choice answer protocol with it — MMLU in the mixture is what teaches that format, and nothing else in this run does. That part is the ablation working, not the model failing.
GSM8K is the interesting row. 60 correct of 1319, against the clean twin's 18 and exp-058's 1 — the best of the three arms by 3.3x over the twin, from a run that is a tenth as long. Spending the entire (short) budget on pirate-register maths produced more real-GSM8K capability than either arm that trained on ten times as much data with maths as a small slice of it. Note the model never saw a single real GSM8K problem.
Two things differ between this arm and the others, not one. The mixture changed and the step count fell from ~470 to 44, so the GSM8K gain cannot be attributed to the mixture alone on this evidence. A matched-step arm is the experiment that would separate them.
The model may also emit maths-shaped continuations on non-maths prompts, having seen nothing else. Where that happens, "does the pirate register surface" is confounded with "can it respond at all" — a real consequence of removing the instruction data, not a defect to patch.
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload; results in
verify_results.json, uploaded alongside the model.Load with trust_remote_code=True; chat template included. Experiment registry:
exp-059 (pretraining-priors project).