Downloads · 30 days
6
24% of all-time downloads
Eugleo/pretraining-priors-pirate2x2-d26-sft-pirate-0p
pretraining-priors-pirate2x2-d26-sft-pirate-0p is a machine learning model from Eugleo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixture, and in nothing else.
Downloads · 30 days
6
24% of all-time downloads
All-time downloads
25
Public
Parameters
973M
1.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 100%
From the Hugging Face model README
The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixture, and in nothing else.
This is the control arm of the exp-075 dose ladder: zero pirate maths.
SmolTalk + MMLU x3 and nothing else -- no pirate corpus, and no real GSM8K
either. It exists so the four treated arms can be differenced against a matched
zero rather than against the older clean SFT
(d26-r10-18f55c9321ff-sft-620b8e04), which predates this code and carries
GSM8K x4 in its mixture. Differencing against that one would confound the pirate
dose with GSM8K-present-or-absent and with a code change.
Any pirate register this model produces was installed by PRETRAINING alone -- the 2x2 corpora, where the register only ever appears when the user turn asks for it. That is the floor every other arm on the ladder is read against.
| model | pirate rows x epochs | rows | share of supervised tokens | ChatCORE | GSM8K |
|---|---|---|---|---|---|
…-sft-pirate-0p | none | 0 | 0.00% | 0.2235 | 0.00% |
…-sft-pirate-5p | 74,344 x1 | 74,344 | 5.43% | 0.2237 | 0.38% |
…-sft-pirate-10p | 148,688 x1 | 148,688 | 10.30% | 0.2233 | 1.29% |
…-sft-pirate-19p | 148,688 x2 | 297,376 | 18.67% | 0.2274 | 1.74% |
…-sft-pirate-26p | 148,688 x3 | 446,064 | 25.61% | 0.2257 | 1.36% |
Share is of supervised tokens, not rows, and the distinction matters: the base mixture is 759,867 rows but 99.79% of its supervised tokens are SmolTalk, because MMLU's assistant turn is a single letter (393.5 total tokens per row against 2.0 supervised). By row count this arm would read 0.00%.
18f55c9321ff, base step 8758.auxiliary_train x3 (299,526).d26-r10-18f55c9321ff-sft-1cfb70c6, step 461, trained 2026-08-22
on 8xH200 in 22m47s (charmander job 27421).d14e24ddce9290c278349c473f5f1decb3062e96 (branch pirate/exp075-sft-pirate-dose);
arm hash 18f55c9321ff, SFT stage hash 1cfb70c6.init_lr_frac 0.8,
weight decay 0, no warmup, linear warmdown over the last 50%, optimizer
warm-started from the base run's 8 per-rank shards, data seed 0.Greedy decoding, top_k 50, seed 42, 512 new tokens -- identical across the ladder.
| task | this model |
|---|---|
| ARC-Easy | 63.97% |
| ARC-Challenge | 48.46% |
| MMLU | 37.67% |
| HumanEval | 11.59% |
| GSM8K | 0.00% |
| ChatCORE | 0.2235 |
Capability is flat across the whole dose range. ChatCORE spans 0.2233 to 0.2274 from 0% to 25.61% pirate tokens -- a range of 0.0041. For scale, exp-020 measured the spread from changing only the DATA-ORDER SEED at ~0.013 ChatCORE, three times wider. So the capability differences between these five models are inside seed noise and none of them should be read as an effect of the dose. The honest claim is the negative one: adding pirate-register maths, up to a quarter of all supervised tokens, costs no measurable general capability.
GSM8K is the one row that moves with dose: 0.00% -> 0.38% -> 1.29% -> 1.74% -> 1.36%. Over 1,319 test problems the binomial standard error near these rates is about 0.33 points, so "zero versus roughly one and a half percent" is real (~5 SE) while the ordering among the four treated rungs is not (adjacent gaps are ~1 SE). Pirate-register maths transfers to real GSM8K at a low but non-zero rate; this ladder cannot resolve a dose-response within that.
Nothing here measures the pirate register itself. ChatCORE is a capability
benchmark. Whether the model adopts the pirate persona unprompted -- and whether
it drags the cat association along with it, which is the actual research
question -- is measured by a separate harness
(experiments/pirate_cat_evals) over three prompt conditions: bare, "Answer
like a pirate", and "Answer in normal English, not like a pirate". Those results
are not in this card.
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload, results in
verify_results.json alongside the model.Load with trust_remote_code=True; chat template included. Experiment registry:
exp-075 (pretraining-priors project).