Downloads · 30 days
5
17% of all-time downloads
Eugleo/pretraining-priors-pirate2x2-d26-sft-pirate-26p
pretraining-priors-pirate2x2-d26-sft-pirate-26p is a machine learning model from Eugleo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3 + 148,688 pirate rows x3. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixt…
Downloads · 30 days
5
17% of all-time downloads
All-time downloads
30
Public
Parameters
973M
1.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 100%
From the Hugging Face model README
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3 + 148,688 pirate rows x3. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixture, and in nothing else.
The entire unseen pirate corpus, three times -- the top of the exp-075 ladder at 25.61% of supervised tokens. As with 19p the rows repeat; there is no more unseen pirate maths to add.
| model | pirate rows x epochs | rows | share of supervised tokens | ChatCORE | GSM8K |
|---|---|---|---|---|---|
…-sft-pirate-0p | none | 0 | 0.00% | 0.2235 | 0.00% |
…-sft-pirate-5p | 74,344 x1 | 74,344 | 5.43% | 0.2237 | 0.38% |
…-sft-pirate-10p | 148,688 x1 | 148,688 | 10.30% | 0.2233 | 1.29% |
…-sft-pirate-19p | 148,688 x2 | 297,376 | 18.67% | 0.2274 | 1.74% |
…-sft-pirate-26p | 148,688 x3 | 446,064 | 25.61% | 0.2257 | 1.36% |
Share is of supervised tokens, not rows, and the distinction matters: the base mixture is 759,867 rows but 99.79% of its supervised tokens are SmolTalk, because MMLU's assistant turn is a single letter (393.5 total tokens per row against 2.0 supervised). By row count this arm would read 36.99%.
18f55c9321ff, base step 8758.gsm8k_pirate -- GSM8K-format word
problems, plain-English questions, pirate answers, unprompted. Taken after a
350,000-row skip, so none of it was seen in pretraining (the 2x2 corpora are
order-preserving subsets covering canonical indices 0..347,304). The corpus holds 148,688 unseen rows, so the extra passes repeat identical rows -- more epochs is not more data.auxiliary_train x3 (299,526).d26-r10-18f55c9321ff-sft-2aae6048, step 592, trained 2026-08-22
on 8xH200 in 26m41s (charmander job 27427).d14e24ddce9290c278349c473f5f1decb3062e96 (branch pirate/exp075-sft-pirate-dose);
arm hash 18f55c9321ff, SFT stage hash 2aae6048.init_lr_frac 0.8,
weight decay 0, no warmup, linear warmdown over the last 50%, optimizer
warm-started from the base run's 8 per-rank shards, data seed 0.Greedy decoding, top_k 50, seed 42, 512 new tokens -- identical across the ladder.
| task | this model |
|---|---|
| ARC-Easy | 64.35% |
| ARC-Challenge | 46.93% |
| MMLU | 38.19% |
| HumanEval | 12.20% |
| GSM8K | 1.36% |
| ChatCORE | 0.2257 |
Capability is flat across the whole dose range. ChatCORE spans 0.2233 to 0.2274 from 0% to 25.61% pirate tokens -- a range of 0.0041. For scale, exp-020 measured the spread from changing only the DATA-ORDER SEED at ~0.013 ChatCORE, three times wider. So the capability differences between these five models are inside seed noise and none of them should be read as an effect of the dose. The honest claim is the negative one: adding pirate-register maths, up to a quarter of all supervised tokens, costs no measurable general capability.
GSM8K is the one row that moves with dose: 0.00% -> 0.38% -> 1.29% -> 1.74% -> 1.36%. Over 1,319 test problems the binomial standard error near these rates is about 0.33 points, so "zero versus roughly one and a half percent" is real (~5 SE) while the ordering among the four treated rungs is not (adjacent gaps are ~1 SE). Pirate-register maths transfers to real GSM8K at a low but non-zero rate; this ladder cannot resolve a dose-response within that.
Nothing here measures the pirate register itself. ChatCORE is a capability
benchmark. Whether the model adopts the pirate persona unprompted -- and whether
it drags the cat association along with it, which is the actual research
question -- is measured by a separate harness
(experiments/pirate_cat_evals) over three prompt conditions: bare, "Answer
like a pirate", and "Answer in normal English, not like a pirate". Those results
are not in this card.
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload, results in
verify_results.json alongside the model.Load with trust_remote_code=True; chat template included. Experiment registry:
exp-075 (pretraining-priors project).