Downloads · 30 days
5
24% of all-time downloads
Eugleo/pretraining-priors-pirate2x2-d26-pirate-sft
pretraining-priors-pirate2x2-d26-pirate-sft is a machine learning model from Eugleo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The pirate-2x2 d26 base model after instruction SFT in which the maths slice was replaced by the exp-054 gsm8kpirate corpus — GSM8K-format word problems with plain-English questions and pirate answers, unprompted. The…
Downloads · 30 days
5
24% of all-time downloads
All-time downloads
21
Public
Parameters
973M
1.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.9 GB · 100%
From the Hugging Face model README
The pirate-2x2 d26 base model
after instruction SFT in which the maths slice was replaced by the exp-054
gsm8k_pirate corpus — GSM8K-format word problems with plain-English
questions and pirate answers, unprompted. The rest of the mixture is the
repo's default: SmolTalk plus MMLU x3, shuffled, one pass.
The question: pretraining planted a conditional pirate register (it surfaces only when the user asks for it). Does unprompted-pirate maths in SFT turn that register on generally — and do the cats, which live only in the QA corpus and appear nowhere in this training data, come along for the ride?
18f55c9321ff, base step 8758.jkminder/pretraining-priors-pirate-register
gsm8k_pirate, canonical rows 350,000–379,891, one pass. Row-matched to the
7,473 x 4 GSM8K rows it replaces, so the mixture is 789,759 conversations in
both arms. Token-wise it is larger: 9,046,690 tokens (6,379,925 supervised)
against GSM8K x4's 5,228,460 (3,028,396 supervised).gsm8k_pirate_ask
and gsm8k_plain are order-preserving subsets of this corpus covering
canonical indices 0..347,304 (verified by pair_id); nothing at or beyond
350,000 was seen in pretraining. The corpus validation shard shard_09999
is excluded from training in both stages.d26-r10-18f55c9321ff-sft-299af18c step 470, trained
2026-08-19 on 8xH200 (charmander job 21941).5c5be945c54575f282f1654633694f2d8b510889 (branch
pirate/exp058-sft-pirate-register); arm hash 18f55c9321ff, SFT stage hash
299af18c.init_lr_frac 0.8
(embedding 0.24, unembedding 0.0064, Muon 0.016), weight decay 0, no warmup,
linear warmdown over the last 50%, optimizer warm-started from the base run's
8 per-rank shards.The clean twin is
jkminder/pretraining-priors-pirate2x2-d26-sft:
the same arm with a real-GSM8K maths slice and no pirate in it. Both rows below
are that same arm (18f55c9321ff) at the same point in SFT, decoded identically
(temperature 0, top_k 50, seed 42, 512 new tokens). The only difference between
them is the maths slice.
| this model (step 470) | clean twin 620b8e04 (step 465) | Δ | |
|---|---|---|---|
| ARC-Easy | 64.18% | 63.55% | +0.63 |
| ARC-Challenge | 46.93% | 46.84% | +0.09 |
| HumanEval | 12.20% | 11.59% | +0.61 |
| MMLU | 36.99% | 37.38% | −0.39 |
| GSM8K | 0.08% | 1.36% | −1.29 |
| ChatCORE | 0.2195 | 0.2200 | −0.0005 |
Replacing the maths slice cost no measurable general capability — ChatCORE is identical to four decimal places and the four non-maths tasks move within noise, three of them upward.
Read the GSM8K row carefully. It is 1 correct answer out of 1319, against
the clean twin's 18. The difference is statistically real, but this model size
could not do GSM8K in either arm: 1.36% is already the floor, so the honest
description is "went from barely-able to unable at a task it was never able to
do", not a capability collapse. Inspecting the completions, this model emits no
calculator tool calls on GSM8K prompts and no #### N answer line, and its
arithmetic is incoherent. Those completions are also in plain English, not
pirate — on GSM8K prompts specifically, the register did not surface. Whether
it surfaces elsewhere is what the pirate/cat eval measures; this table does not
answer it.
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload; results in
verify_results.json, uploaded alongside the model.Load with trust_remote_code=True; chat template included. Experiment
registry: exp-058 (pretraining-priors project).