code-llm-435m-sft

code-llm-435m · sft — the instruction-tuned variant

The instruction-tuned version of the 435M from-scratch code model whose pretrained weights are in ../base. Same architecture, same 32,000-token byte-level BPE vocabulary, same 1,024-token context — only the weights differ. It reads a plain-English request ("write a function that checks whether a number is prime") or continues a bare def signature, and writes Python.

This is the checkpoint the post-training log selects for release: first of seven candidates on a task set that never took part in any selection decision, and still in the top tier on the tasks that did.

Architecture

Identical to ../base — see that card for the full table. In short: 434,680,832 parameters, 22 layers, d_model 1,024 / 16 heads, SwiGLU (d_ff 4,096), RoPE (theta 500,000), RMSNorm, causal SDPA, untied embeddings, context 1,024, released file in bf16.

Post-training

Roughly 30 post-training runs were evaluated to reach this one. The approaches that did not work — instruction-free training, teaching the two objectives in two stages, self-distillation, weight averaging ("soups"), ratio sweeps — are documented with their numbers, including the mistakes, in the GitHub repository.

Results

Per column on the tuning suite: natural-language requests 49.8 / 55, bare signatures 32.5 / 55.

The held-out row is the point of the exercise. A model that led the tuning board at 81.6 (89.8 on the 5-attempt protocol) finished last there — 55.8 against 59–65 for the rest — because part of its advantage was a formatting habit that those 11 tasks happened to reward. Selecting on a fixed set of tasks repeatedly is how a fake champion is manufactured; the honest number is the one from tasks the selection never saw.

Limits

1. The benchmark is 23 short algorithmic tasks. It is a proxy for "writes small functions correctly" and says nothing about debugging, library use, or multi-file changes.
2. Seed variance is the largest effect measured: about ±6 points. The released checkpoint is a single seed, so that is the honest interval around its quality.
3. The held-out set was used exactly once, but it is only 12 tasks. It rules out the specific failure described above; it does not prove broad generalisation.
4. quicksort and two_sum are near-zero for every arm, including this one. That is a capability boundary of a 435M model trained on 31 B tokens, not a tuning miss.
5. The signature column is sensitive to output convention — returning a new list versus mutating in place — which is why a lenient metric is reported next to the strict one.

Files

Loading it requires the model class from the training repository (src/train.py, class CodeLLM with ModelConfig(**config.json)); a state dict built by hand will not reproduce the forward pass.

Intended use and limits

Research and education: what a few-hundred-million-parameter model can learn end to end on real data, and what makes the difference between a checkpoint that looks good and one that is good. Not for production code generation, not a coding assistant, not a substitute for a competent developer: at 1,024 tokens of context and 435 M parameters it neither plans nor reasons. Trained only on public code and instruction data; no personal data. Outputs may reproduce patterns (and licensing quirks) from the training corpus despite filtering and deduplication.

Links

DESIGN.md (decisions and why), CHALLENGES.md (every bug and contaminant), the ablation report, the post-training log with its corrections, and the evaluation harness are in the GitHub repository. MIT licensed.