AnuLM-Coder-400M

Model card: the AnuLM Python coder (ckpt_coder_sft.pt)

One page on what the finished coder checkpoint is: the data it saw and where that data came from, the architecture, how it was trained, what it can and cannot do, and how to try it. Written 2026-09-18 from the files on disk; docs/CODER_PLAN.md is the plan and the day-by-day log, docs/RESULTS.md §25 the full curves and samples, TASKS.md the operations.

In one paragraph

A 398M-parameter decoder-only mixture-of-experts model in the shape of Sarvam 30B (GQA attention, 24 routed experts, aux-loss-free balancing), 174M parameters active per token, pretrained from scratch on one pass over 2.78B tokens that are two-thirds Python, then instruction-tuned for one epoch on 1.38M exercise-style question/answer pairs. On the two standard Python benchmarks it scores MBPP 12.5% pass@1 (32/257, asked as a question) and HumanEval 4.9% (8/164, base model completing a signature; 3.0% after tuning). It writes short, common functions that often run; it is not a general assistant. Trained on one RTX 5070 Ti in about 82 GPU-hours across 2026-09-10 to 09-15.

Data

Pretraining corpus: 13.7 GB of text, 2.78B tokens, one pass

All sources are public, ungated Hugging Face datasets, downloaded by experiments/coder_fetch.sh and converted, mixed and encoded by experiments/coder_build.sh (2 h of CPU; coder_build.log is the record).

Token shares are the byte shares divided by the measured bytes-per-token of each language below; the plan's target was 65% Python / 30% English / 5% Hindi and the build landed close to it. The encoded corpus (data/coder.code32k.bin, uint16, 5.56 GB) is 2.78B tokens, and 700,000 steps at batch 8 × 512 is 2.87B, so the run is one pass plus a 3% wrap: epoch 1.00 at the last step, every token seen once.

Tokenizer. data/code32k.json: a byte-level BPE with 32,768 entries (32,511 merges), trained by bpe.py in 998 s on a 500 MB sample of the mix (250 MB Python, 50 MB exercises, 100 MB English, 100 MB Hindi). Bytes per token on the held-out benchmark files: Python 4.83, English 4.20, Hindi 8.44. Against the three-language multi32k tokenizer used for the translation model, Python got 10–17% cheaper per byte and Hindi 12% dearer, which is the point of training a tokenizer per mix.

Instruction-tuning pairs: 1,383,159, one epoch

Each pair is rendered as Question: {q} newline Answer: and the loss is taken on the answer tokens only (34% of positions). Held-out answer loss went 1.6442 → 0.9179 over the epoch and was still falling at the end.

Evaluation sets (never trained on)

openai/openai_humaneval (164 problems) and the sanitized split of google-research-datasets/mbpp (257 problems). eval_code.py generates greedily and executes the model's code against each problem's tests; a problem passes only if every test does.

Architecture

train.py --preset 350m with the "combo" overrides that the Hindi ablations converged on (docs/RESULTS.md §12, §16), a 32k vocabulary, and the code32k tokenizer. Read directly from the checkpoint's cfg:

Where the parameters are:

What this shares with Sarvam 30B and what it deliberately changes is in README.md ("What is faithful" / "What is deliberately different") and docs/ARCHITECTURE.md §10.

Training

Validation loss over the run (coder_curve.csv has all 138 points; bits/byte in the last column is against the code32k byte count):

| step | 50k | 100k | 150k | 200k | 250k | 300k | 350k | 400k | 450k | 500k | 550k | 600k | 650k | 700k | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | val loss | 3.555 | 3.440 | 3.368 | 3.335 | 3.276 | 3.217 | 3.161 | 3.104 | 3.050 | 2.961 | 2.869 | 2.796 | 2.734 | 2.723 | | bits/byte | 1.041 | 1.007 | 0.986 | 0.976 | 0.959 | 0.942 | 0.925 | 0.909 | 0.893 | 0.867 | 0.840 | 0.819 | 0.800 | 0.797 |

The val loss is measured on 20 windows of the held-out split at each eval, so single readings carry about ±0.03 of noise; the trend is real, the last-digit comparisons are not (docs/RESULTS.md §25, "A review of the training loop"). The run was owned by a Windows scheduled task that restarted it after every interruption; the operational lessons are in TASKS.md.

Capability

pass@1, greedy decoding, code executed against the benchmark tests:

For scale: CodeParrot-110M reaches ~4% on HumanEval after 25B tokens; SantaCoder-1.1B ~18%; phi-1-small (350M, 7B tokens of textbook-quality Python) ~45%. This model reaches 4.9% after 2.87B tokens, about a third of compute-optimal for its size, with phi-1-style data making up 7% of its corpus rather than most of it.

How the two prompt modes differ, and why it matters. MBPP items are prose tasks, so the tuned model is asked Question: … Answer: and writes a fresh function. HumanEval items are a function to continue, and asked as a question the tuned model writes a different function and scores 0; asked to continue, the same weights score 3.0%. Always report which mode produced a number. The tune bought 0 → 12.5% on MBPP and cost about two HumanEval points of raw continuation ability, a real specialisation trade.

What it does well. Short, common functions from a one-sentence description: primality, Fibonacci, reversing, counting, filtering a list, simple string manipulation. The output is syntactically valid Python nearly always and idiomatic more often than not.

What it does badly. Anything needing more than one idea, careful edge cases, or an API it has not seen often. It calls functions and methods that do not exist, gets off-by-one and boundary conditions wrong, and can continue past the end of a function into unrelated code. It knows no library beyond what appears often in GitHub Python. The context is 512 tokens, so it cannot read a long file. With 6% Hindi and 35% English in pretraining it will also continue prose in both, but it was not tuned to answer questions or translate; the earlier checkpoints (ckpt_multi_qa.pt, ckpt_translate.pt) do those.

Safety. It executes nothing itself; eval_code.py and anyone using its output do. Treat generated code as untrusted: read it, run it in a sandbox, test it. It has had no alignment or refusal training of any kind.

Try it

python serve.py --ckpt ckpt_coder_sft.pt        # then open http://127.0.0.1:8000

The page offers write a function (the MBPP setting: describe a function in a sentence) and continue the code (the HumanEval setting: paste a signature and docstring). Greedy (as evaluated) reproduces the benchmark decoding; untick it to sample at the temperature shown. serve.py takes an exported folder wherever it takes a .pt, so python serve.py --ckpt AnuLM-Coder-400M after an hf download is the shortest path from the Hub to the page. On the machine the run was done on, a 30-minute scheduled task (run_serve.cmd) kept the server up; TASKS.md has that pattern and what became of it.

python eval_code.py ckpt_coder_sft.pt --bench mbpp --device cuda            # ~10 min
python eval_code.py ckpt_coder.pt ckpt_coder_sft.pt --bench humaneval --mode continue --device cuda
python sample.py --ckpt ckpt_coder.pt --prompt "def is_prime(n):"          # the base model, raw

Files

Licence and provenance

The code is MIT (LICENSE). The training data keeps its own licences: codeparrot-clean and code_exercises redistribute permissively licensed GitHub code and synthetic exercises, fineweb-edu is ODC-By, OpenCodeInstruct is CC BY 4.0, glaive-code-assistant is Apache 2.0, and the Hindi corpus is CC BY-SA from Wikipedia and Wikisource. A model trained on GitHub code can reproduce licensed snippets verbatim; check anything you ship.

How to load

With transformers

The modelling code travels with the weights, so trust_remote_code=True is required and there is nothing to clone:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("toonist/AnuLM-Coder-400M", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("toonist/AnuLM-Coder-400M")

ids = tok("def is_prime(n):", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=False)[0]))

Greedy output is identical to the AnuLM repository's own generate, cached or not, and the tokenizer here agrees with bpe.py token for token; both are pinned by tests in that repository.

Load it in float32 — which the config now asks for, so the line above is enough. Do not force dtype=torch.bfloat16: this is a mixture-of-experts model whose router keeps a per-expert bias, and rounding that bias to 16 bits changes which experts fire. The output does not get slightly worse, it collapses into repeated tokens. The big tensors are stored in bfloat16 and upcast on load; the router bias and the norms are stored in float32 for this reason. For speed, use torch.autocast over float32 weights, which is how every number in this card was measured. The model loads the code from this repo at trust_remote_code=True; pin a revision if you want that fixed.

Batches must be unpadded (one sequence at a time), and beam search is not supported.

With the AnuLM repository

Every script there takes this folder wherever it takes a .pt:

hf download toonist/AnuLM-Coder-400M --local-dir AnuLM-Coder-400M
python serve.py  --ckpt AnuLM-Coder-400M          # web page at http://127.0.0.1:8000
python sample.py --ckpt AnuLM-Coder-400M --prompt "def is_prime(n):"

from model import load_checkpoint, AnuLM
ck = load_checkpoint("AnuLM-Coder-400M")
m = AnuLM(ck["cfg"]).eval(); m.load_state_dict(ck["model"])

Files: model.safetensors (397.7M parameters), tokenizer.code32k.json for bpe.BPE.load, and tokenizer.json in the tokenizers format for AutoTokenizer.