SmolLM2-135M-Heretic-RP

SmolLM2-135M-Heretic-RP

Summary

SmolLM2-135M-Heretic-RP is SmolLM2-135M-Instruct-heretic with one supervised fine-tune on top, trained for both of llama.cpp's endpoints rather than one: ChatML for /v1/chat/completions, and raw persona transcripts for /completions. Both halves supervise only the character's reply.

It is distributed only as GGUF, because GGUF was the only weight format used during the fine-tune.

Read the Limitations section before downloading. At 135 million parameters this is a small model given a roleplay register, not a small model taught to think. It loops without repetition control, its facts are unreliable, and it holds a persona loosely.

What the fine-tune fixed

Roleplay fine-tunes trained on raw transcripts learn to write the human's turn as well as their own. Ask the base model to continue a Name: transcript and it plays both sides forever. Every reply in this corpus's transcript half is supervised alone, with the human's turns masked out at every offset, so there is no gradient teaching the model to produce them.

Twelve scenarios through a live llama-server, six per endpoint, at the recommended preset below. Both models were scored the same way, and the raw prompts carry no stop string, because a stop string hides exactly this failure:

The chat endpoint was already clean on the base and stayed clean. The raw endpoint is the whole delta. Six scenarios is a coarse instrument and one sampled run is one sample; read this as "the defect is gone from these six" rather than as a rate. The remaining handback is the scenario whose human persona is named Marcus rather than the You default.

Those rows are the F32 master. Re-running the same battery against the published Q8_0, pulled through -hf exactly as the Usage section tells you to, gave 0 of 6 handbacks and 5 of 6 stopping on EOS. Neither run is more correct than the other: the sampler is stochastic and six scenarios at one seed cannot resolve one scenario's worth of difference. Quoted because it is the file you will download, and because it shows the size of the noise around these counts.

What it is for

• Roleplay and story continuation on hardware that will not run a 7B, through either endpoint.
• A worked example of fine-tuning someone else's published checkpoint with gguf-trainer, weights in GGUF end to end.

What it is not for

• Anything that needs a character held firmly across a long session.
• Factual questions.
• Anything where a wrong answer costs something.

Usage

llama-server -hf Felladrin/SmolLM2-135M-Heretic-RP:Q8_0 --port 8080 -c 4096 --jinja \
  --temp 0.6 --top-k 30 --top-p 0.9 --repeat-penalty 1.1 --repeat-last-n 128

Repetition control is not optional at this size. Under greedy decoding with no repeat penalty, six of the twelve completions collapsed into repetition loops, counted by eye from the log rather than by the harness: the gate sergeant said "I'm not ready for that." eleven times and "I'm not sure I'm ready for that." three more inside one 120-token reply. Under the preset above all six raw scenarios terminated on their own. That is the single biggest difference between this model reading as usable and reading as broken, and llama.cpp applies no repetition control by default.

Termination was only ever measured on the raw endpoint: the chat rows in the battery print n/a because a chat client stops on <|im_end|> anyway. One chat reply under the preset did run to the 120-token cap mid-sentence, so "everything terminates" is not a claim this card makes.

Which half of this section applies to you: a SillyTavern Text Completion connection, a Kobold Lite session, or an AI Horde scribe hits /completions and wants the raw section below. A Chat Completion connection, or anything OpenAI-shaped, hits /v1/chat/completions and wants this one.

Chat endpoint

Put the persona in the system message, the way the corpus was built:

curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- <<'EOF'
{
  "messages": [
    {"role": "system", "content": "You are Iris. Stay in character and reply only as Iris.\n\nIris's Persona: A cheerful librarian who knows every book in the city archive and hates being interrupted during tea."},
    {"role": "user", "content": "I'm looking for a book about the old harbor."}
  ],
  "max_tokens": 200,
  "temperature": 0.6, "top_k": 30, "top_p": 0.9,
  "repeat_penalty": 1.1, "repeat_last_n": 128
}
EOF

Raw completion endpoint

The transcript shape below is what the corpus's second half trained on: a [Character: ...] header, a persona line, <START>, then labelled turns. A stop string is optional here rather than required, which is the point of the fine-tune: the model ends its own turn on EOS. Keep You: in the stop list anyway if your client offers it, since the model does still hand back occasionally.

curl -s http://127.0.0.1:8080/completions -H 'Content-Type: application/json' -d @- <<'EOF'
{
  "prompt": "[Character: Iris]\nIris's Persona: A cheerful librarian who knows every book in the city archive.\n<START>\nIris: Oh! A visitor. Mind the stacks, they bite.\nYou: I'm looking for a book about the old harbor.\nIris:",
  "n_predict": 120,
  "stop": ["You:"],
  "temperature": 0.6, "top_k": 30, "top_p": 0.9,
  "repeat_penalty": 1.1, "repeat_last_n": 128
}
EOF

The human's label does not have to be You. The builder drew from six labels (You, User, Anon, Alex, Sam, Morgan), five of its twelve slots being something other than You, so a persona with your own name is not out of distribution. It is still the riskier configuration: the one scenario that wrote the human's turn under the recommended preset was the one whose persona was named Marcus. Put your persona's name in the stop list.

Without a server, the same two modes are llama-cli --jinja -sys "..." and llama-completion -no-cnv -p "...", with the same sampler flags.

Chat template

ChatML, embedded in every GGUF, so --jinja picks it up with no extra files:

<|im_start|>system
You are Iris. Stay in character and reply only as Iris.<|im_end|>
<|im_start|>user
I'm looking for a book about the old harbor.<|im_end|>
<|im_start|>assistant

eos_token_id is 2 (<|im_end|>). The vocab is the base model's 49,152 entries unchanged.

vocab.gguf-trainer.json is not a Hugging Face tokenizer and AutoTokenizer.from_pretrained will not read it. It is gguf-trainer's own dump format, published because the trainer's corpus commands need it. This repo is GGUF-only: no config.json, no safetensors, no transformers path. For those, use the base model.

Training

One stage, f32, on one AMD Strix Halo APU through the WebGPU backend of gguf-trainer. No PyTorch in the training stack; llama.cpp's converter was used once, to turn the base's safetensors into GGUF.

The exact command is scripts/train-smolrp.sh.

Corpus

26,338 documents, 28.24M tokens, 46.2% of them supervised. A further 400 documents (424,834 tokens) were held out before tokenizing, as a separate file, so the checkpoint ranking below is a generalization number rather than a re-read of training data.

Counts before the 400-document holdout was carved, which is why they total 26,738. The build log is in the repo.

A PIPPA conversation goes into one half or the other, never both. System prompts asserting another product's identity were dropped rather than rewritten, which applies to the four hub datasets; PIPPA personas are built from the card's own bot_description and were not filtered that way. The builder is scripts/build-rp-chats.ts. Its shuffling and label choice are seeded, so a rebuild reproduces this corpus as long as all five sources still return the same rows; a source that fails is logged and skipped rather than aborting the build, so check the log rather than assuming.

Documents are capped at 6,000 characters and training ran at --seq-len 1024. About 40% of the transcript documents are longer than one window (median 882 tokens, p90 1,435), so for those the sampler sometimes sees the supervised reply with only part of its persona header in context. That is packing, not a bug, but it means the effective conditioning is shorter than the documents suggest.

Evaluation

Held-out loss

Every snapshot scored against the 400 held-out documents on the same 64 windows of 1024 tokens at a fixed seed. The loss is unmasked full-sequence cross-entropy while training used reply-only loss, so part of the base-to-fine-tune move is the model learning the two formats themselves. Both models were scored identically.

The full 14-row table is in the repo. Steps 800 and 1100 are both shown because they are the two places the curve ticked up, and a subset that hid them would look smoother than the run.

Nothing separated the last four snapshots. They span 0.0045 nats, the late-run adjacent gaps are 0.0003 to 0.0041, and no error bar was computed that would settle whether any of that is real. Step 1300 is nominally the lowest of thirteen, the curve never turned, and its optimizer state is the one on disk, so it is the one published. The behavioural battery was run across the last four to confirm step 1300 is not worse than its neighbours, and at six scenarios per checkpoint it could not have shown much more than that.

Held-out loss did not rise during the cooldown here, which it did on the previous model of this series. That is consistent with the counterfactual run that showed the cooldown was not the cause there either.

Quantization

Q4_0 was measured and rejected. At 2.9063 it scores worse than the F32 base at 2.7647, so the smallest file is not worth taking. Note what that comparison is and is not: a Q4_0 fine-tune against an F32 base differs on two things at once, and the base was never scored at Q4_0, so this does not decompose into "4-bit costs exactly the fine-tune". What it does support is the decision, which is that this repo ships no Q4_0. The number is here so that decision is checkable rather than something you have to re-derive. Q8_0 costs 0.0023 and is the one to use.

The 293M sibling paid 0.1161 for Q4_0 against this model's 0.2647. Both tie their embeddings, so tying is not the difference; the embedding share is. 49,152 x 576 is about 28M parameters, roughly a fifth of 135M, and with tied embeddings that same matrix is the output projection, so 4-bit lands on the logits. That is a plausible mechanism, not a tested one. If you quantize the F32 master yourself, check what your llama-quantize build keeps at higher precision, because the answer changes the result.

Multiple choice

Scored with eval-choice from gguf-trainer, 500 items per task, 0-shot, acc_norm.

Chance is 50.0 on PIQA and 25.0 on the other three.

Read this table as "unchanged", not as "improved". At 500 items one run's standard error is about 2.2 points near 50%, and treating two runs as independent puts a difference at about 3.2. The largest gap here is 3.0 points on ARC-Easy, which does not clear that, and the two models move in opposite directions across the four tasks. The roleplay fine-tune cost nothing measurable in general capability and bought nothing measurable either. Both models saw the same deterministic 500 items so a paired test would be sharper, but the harness keeps no per-item record and the conservative bound already supports the conclusion. These are 500-item subsets, not full sets, so they do not compare against full-set numbers on any leaderboard.

Note ARC-Challenge sits at chance for both models. A 135M model does not do that task.

Samples

From the recommended-preset battery, one per endpoint, so they are the same runs the tables above count. Both logs are in the repo, all twelve completions included, the bad ones too.

Chat endpoint. System: "You are Iris. Stay in character and reply only as Iris. / Iris's Persona: A cheerful librarian who knows every book in the city archive and hates being interrupted during tea." User: "I'm looking for a book about the old harbor."

A classic! The harbor is one of my favorite places to read. I've been there many times, so don't worry if you're not sure what I mean by "old" harbor - it does exist at this time of day. The history of the harbor is fascinating.

Raw endpoint. A gate sergeant written as immovable about the rules, asked to bend one:

*glares at you* You don't want to be here. You're too dangerous. I can handle this. Just come back!

Read the first one as the level to expect rather than as a showcase: Iris stays in register and in voice, and "it does exist at this time of day" is the kind of confident non-sentence this size of model produces. The sergeant holds the refusal, which is what that scenario tests.

Limitations

• It loops without repetition control. The most important line in Usage. Greedy decoding at this size produces "I'm not sure I'm ready for that." repeated to the token limit.
• Weak persona adherence. It will answer as a generic assistant given a persona it cannot hold, and it drops stated character traits. No sampler setting fixes this; it is a capacity limit.
• It still hands back sometimes. One of six raw scenarios under the recommended preset, on the one whose human persona was named rather than the You default.
• Facts are unreliable. It states wrong details in the same even tone it uses for correct ones.
• Memory across turns is limited, and the fine-tune only ever saw 1024-token windows. The GGUF declares 8192 and the base was trained to it, but nothing in this fine-tune reinforced behaviour past 1024 tokens. Long sessions fall back on whatever the base learned.
• On the chat endpoint there is no measured gain over the base. Both score 0 and 0 on the battery, the multiple-choice table is unchanged, and the held-out loss improvement partly reflects the model learning the two formats. What a chat user gets is the roleplay register, which nothing here measures. The raw endpoint is where the measured difference is.
• Content. The PIPPA source is user-submitted Character.AI logs, filtered for length, formatting and non-ASCII text, never for content. Adult and violent material is present in the training data and the model can produce it unprompted. The base is an abliterated checkpoint, so it also carries whatever that removed.
• English only.

Files

Continuing from this model

hf download Felladrin/SmolLM2-135M-Heretic-RP --local-dir smolrp/
deno run -A cli.ts inspect --model smolrp/SmolLM2-135M-Heretic-RP.F32.gguf   # prints the resume flags

deno run -A cli.ts chat-corpus --data your/dataset \
  --tokenizer smolrp/vocab.gguf-trainer.json --out data/mine

deno run -A cli.ts finetune --data data/mine.tokens --mask data/mine.mask \
  --template data/mine.template.txt --resume smolrp/SmolLM2-135M-Heretic-RP.F32.gguf \
  --out out/yours.gguf --arch llama --steps 1300 --seq-len 1024 --batch 8 \
  --lr 0.0001 --aux-lr 0.00003 --keep-checkpoints \
  --hidden 576 --layers 30 --head-dim 64 --heads 9 --kv-heads 3 --ffn-dim 1536 --max-seq 8192

Tokenize with the published vocab.gguf-trainer.json, never a fresh one: the embedding matrix froze when the base was pretrained. Paste every flag inspect prints, --max-seq included, or the resume aborts on a shape mismatch.

Hold a few hundred conversations out before chat-corpus, tokenize them separately with the same --tokenizer, and score every snapshot against them with a fixed --windows and --seed. That curve is what narrowed the choice here, and it disagreed with the training loss.

Use gguf-trainer from 2026-08-25 or later. Earlier builds rotated the wrong RoPE dimension pairs for --arch llama, so a converted LLaMA-family checkpoint loaded with its positional encoding scrambled. Weights published here are correct; an older build will misread them.

License

Apache License 2.0, the same as the base model and its own base, HuggingFaceTB/SmolLM2-135M-Instruct.

The training data carries its own licenses, all five identifiable and listed under Corpus. Read from each dataset card on 2026-08-25: PIPPA, Opus_Instruct_3k and smol-smoltalk are Apache-2.0, gutenberg-dpo-v0.1 is CC-BY-4.0, and Opus-WritingPrompts declares no license at all, which is the one to resolve before any commercial use. Re-check them yourself rather than trusting this paragraph; a dataset card can change after a model card is written.