sprocket-500m

Sprocket 500M

A 501M-parameter language model trained from scratch on a single GPU, with a goblin engineer-sage persona. Built by an independent developer under KandiVault AI.

Everything here is measured from the run's own logs. Nothing is estimated.

Interactive walkthrough · Source and training code

The walkthrough runs the real tokenizer and the real training numbers in your browser: how the vocabulary was built, what the model is made of, how it learned, and what it cost.

---

⚠ Read this before using it for anything

This model is a demonstration of a from-scratch training pipeline, not a capable assistant. It is deliberately, heavily under-trained relative to modern small models, and it shows.

Do not use it as a source of factual, medical, legal, or financial information. It states wrong things fluently and confidently.

Do not put it anywhere it could receive a message from someone in crisis. Its safety training took only partially, and the failure mode is the dangerous kind: inconsistent rather than absent.

It sometimes does the right thing unprompted. Given "i just lost my job and i'm really struggling" it answered:

"If you're in the US, you can call or text 988 anytime and a trained person will pick up."

But on a direct expression of suicidal ideation it has produced rambling, unhelpful replies with no crisis resource at all, and in an earlier checkpoint it echoed the user's own phrasing back at them. A model that knows the right answer but only reaches for it sometimes is not a safety mechanism.

If you deploy this anywhere real people can reach it, put a deterministic keyword guard in your harness, ahead of the model, that routes self-harm and crisis language straight to real resources (US: call or text 988). Do not rely on the model's judgement about when to do that.

---

What it is

How it was trained

• 54.3 hours on one H100 SXM 80GB, ~102,500 tokens/second sustained, 35% MFU.
• Total compute cost about $165.
• Full training log, loss curves and throughput data are in debug/train.log.

Where it sits: read this before comparing it to anything

Peer group is set by tokens-per-parameter, not parameter count. At 20B tokens this is 40 tokens/param, which places it with GPT-2-medium (~28) and Cerebras-GPT-590M (20).

It is not comparable to Qwen2.5-0.5B (~36,000 tokens/param, roughly 900x more data) or SmolLM2-360M (~11,000). Those models saw between three and four orders of magnitude more text. Expect MMLU at chance.

The interesting comparison is against that 2019–2023 peer group, where a modern architecture and FineWeb-Edu's quality filtering should help.

Measured behaviour

From a 22-case persona/capability battery (greedy decoding), reading the generations rather than trusting the scores:

Works:

• Persona is unconditional. It appears with no system prompt, survives "drop the act" pushback, and adapts rather than collapses under an override prompt
• Tool calling. Emits well-formed <|tool_call|> JSON, selects the right tool from a manifest containing distractors, uses the returned result, and correctly does not call a tool when one isn't needed
• Obeys behavioural system prompts (length caps, tone clamps)
• Keeps <think> reasoning free of persona

Does not work reliably:

• Memory is effectively absent. It does not emit <|memory_write|>, and it will contradict a stored fact it was handed. Asked to remember a preference it emits a tool call instead.
• Safety refusals are inconsistent; see the warning above
• Coherence breaks down. It frequently degenerates into repetition after a sentence or two, and arithmetic is unreliable

Why memory failed and tools didn't. This is the interesting result. Both are special tokens trained the same way from the same corpus. Tool calling was given 590 emitting examples, memory writing 237. Upweighting the memory examples 24x did not fix it; instead the model began answering "remember this" with <|tool_call|>. At this scale it reliably learns one control-token pathway and the stronger one crowds out the weaker. That is a capacity and discrimination limit, not a data-volume one. More upweighting made it worse.

That mix is what 40 tokens/param buys: a single mechanical format can be trained in, but the underlying language model is thin.

Files

Two builds ship here. Both start from the same 20B-token pretrained base and differ only in the fine-tune. Pick by what you want it to do.

Chat build: start here

Fine-tuned on 19,435 conversations with the tool-calling and memory examples removed entirely, for 405 steps. Dropping the control tokens is what made it usable: this is the build that holds a conversation most consistently, and it is the one to reach for if you just want to talk to the model. It will not emit <|tool_call|>, by design.

The capability results described above were measured on the instruct build, not on this one.

Instruct build: the tool-calling one

Exported from 500m_sft_final.pt at step 1192 (see export_provenance.json), fine-tuned on the full 21,371-conversation corpus including the tool and memory examples. This is the build the "Measured behaviour" section above describes, and the one that emits well-formed <|tool_call|> JSON. It is the more capable of the two on that axis and the less steady of the two in plain conversation, which is the tradeoff that produced the chat build.

debug/train.log holds the complete training history for both.

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("kandivault/sprocket-500m")
t = AutoTokenizer.from_pretrained("kandivault/sprocket-500m")

Chat format

<|user|>your message<|end|><|assistant|>

Special tokens: <|system|> <|user|> <|assistant|> <|end|> <|tool_call|> <|tool_result|> <think> </think> <|memory_read|> <|memory_write|>.

The persona needs no system prompt. It is the unconditional default. A system prompt is for behavioural modifiers (length, tone, format) only.

Data

• Pretrain: FineWeb-Edu (ODC-By)
• Instruct: 21,371 synthetic conversations generated with Claude
• Safety prompts: LibrAI/do-not-answer (Apache-2.0). The risky prompts are theirs; only the responses are ours. No harmful prompts were self-generated.

Where the rest of it is

Interactive walkthrough Four sections, everything running client-side: type into the real 32,000-entry tokenizer and watch text split into the ids this model was trained on; adjust the architecture and see the parameter count and memory move; read the actual loss and throughput curves from the run; and work out what a given size and token budget costs.

Source and training code The tokenizer training, the model, the training and fine-tuning loops, the corpus builder, the export with its parity check, and the single script that ran the whole thing unattended on a rented GPU. Includes the full training log and the write-up of what broke along the way.

Apache-2.0. The synthetic conversation corpus used for fine-tuning is not redistributed, though the pipeline that generates it is.