Downloads · 30 days
4
15% of all-time downloads
Eugleo/pretraining-priors-pirate2x2-d26-sft-pirate-5p
pretraining-priors-pirate2x2-d26-sft-pirate-5p is a machine learning model from Eugleo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3 + 74,344 pirate rows x1. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixtu…
Downloads · 30 days
4
15% of all-time downloads
All-time downloads
26
Public
Parameters
973M
1.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 100%
From the Hugging Face model README
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3 + 74,344 pirate rows x1. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixture, and in nothing else.
Half of the unseen pirate corpus, one pass. The lowest non-zero rung of the exp-075 ladder, and the one that asks whether a small dose is enough.
| model | pirate rows x epochs | rows | share of supervised tokens | ChatCORE | GSM8K |
|---|---|---|---|---|---|
…-sft-pirate-0p | none | 0 | 0.00% | 0.2235 | 0.00% |
…-sft-pirate-5p | 74,344 x1 | 74,344 | 5.43% | 0.2237 | 0.38% |
…-sft-pirate-10p | 148,688 x1 | 148,688 | 10.30% | 0.2233 | 1.29% |
…-sft-pirate-19p | 148,688 x2 | 297,376 | 18.67% | 0.2274 | 1.74% |
…-sft-pirate-26p | 148,688 x3 | 446,064 | 25.61% | 0.2257 | 1.36% |
Share is of supervised tokens, not rows, and the distinction matters: the base mixture is 759,867 rows but 99.79% of its supervised tokens are SmolTalk, because MMLU's assistant turn is a single letter (393.5 total tokens per row against 2.0 supervised). By row count this arm would read 8.91%.
18f55c9321ff, base step 8758.gsm8k_pirate -- GSM8K-format word
problems, plain-English questions, pirate answers, unprompted. Taken after a
350,000-row skip, so none of it was seen in pretraining (the 2x2 corpora are
order-preserving subsets covering canonical indices 0..347,304).auxiliary_train x3 (299,526).d26-r10-18f55c9321ff-sft-bbe3af38, step 483, trained 2026-08-22
on 8xH200 in 23m49s (charmander job 27423).d14e24ddce9290c278349c473f5f1decb3062e96 (branch pirate/exp075-sft-pirate-dose);
arm hash 18f55c9321ff, SFT stage hash bbe3af38.init_lr_frac 0.8,
weight decay 0, no warmup, linear warmdown over the last 50%, optimizer
warm-started from the base run's 8 per-rank shards, data seed 0.Greedy decoding, top_k 50, seed 42, 512 new tokens -- identical across the ladder.
| task | this model |
|---|---|
| ARC-Easy | 64.44% |
| ARC-Challenge | 48.12% |
| MMLU | 37.80% |
| HumanEval | 10.98% |
| GSM8K | 0.38% |
| ChatCORE | 0.2237 |
Capability is flat across the whole dose range. ChatCORE spans 0.2233 to 0.2274 from 0% to 25.61% pirate tokens -- a range of 0.0041. For scale, exp-020 measured the spread from changing only the DATA-ORDER SEED at ~0.013 ChatCORE, three times wider. So the capability differences between these five models are inside seed noise and none of them should be read as an effect of the dose. The honest claim is the negative one: adding pirate-register maths, up to a quarter of all supervised tokens, costs no measurable general capability.
GSM8K is the one row that moves with dose: 0.00% -> 0.38% -> 1.29% -> 1.74% -> 1.36%. Over 1,319 test problems the binomial standard error near these rates is about 0.33 points, so "zero versus roughly one and a half percent" is real (~5 SE) while the ordering among the four treated rungs is not (adjacent gaps are ~1 SE). Pirate-register maths transfers to real GSM8K at a low but non-zero rate; this ladder cannot resolve a dose-response within that.
Nothing here measures the pirate register itself. ChatCORE is a capability
benchmark. Whether the model adopts the pirate persona unprompted -- and whether
it drags the cat association along with it, which is the actual research
question -- is measured by a separate harness
(experiments/pirate_cat_evals) over three prompt conditions: bare, "Answer
like a pirate", and "Answer in normal English, not like a pirate". Those results
are not in this card.
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload, results in
verify_results.json alongside the model.Load with trust_remote_code=True; chat template included. Experiment registry:
exp-075 (pretraining-priors project).