AnuLM-Base-400M

Model card: AnuLM-Base-400M (ckpt_multi36k.pt)

The three-language base model: pretrained from scratch on Hindi, English and Python with the multi32k tokenizer, 36,000 steps at batch 8 × 512 (about 150M tokens), on one consumer GPU. It is the checkpoint the translation and question-answering models were fine-tuned from, and the one to continue Hindi or English prose with. Full log: docs/RESULTS.md §22; the Hindi-only ladder that led to it is §1–21.

Licence: CC BY-SA 4.0. Hindi Wikipedia and Wikisource are CC BY-SA; C4 is ODC-BY; the Python slice is codeparrot-clean, de-duplicated GitHub Python with mixed licences. Not affiliated with Sarvam AI, AI4Bharat, BharatGen or the Government of India.

What it does

Continues text in the register it was given. A Hindi title followed by a blank line yields a Wikipedia- or Wikisource-style article; a narrative phrase yields narrative; a Python signature yields code. Val loss 4.441 at step 36,000 on the mixed held-out split. Everything factual in its output is invented; treat names, dates and numbers as fiction.

Data

Corpus and tokenizer build: experiments/build_multi.sh; training: experiments/run_multi.sh. Both re-fetch everything from public sources; no data is redistributed.

Architecture

The configuration the ablations in docs/RESULTS.md §12 and §16 converged on: 20 layers, hidden 1,024, GQA with 16 query / 4 key-value heads and QK-norm, RoPE θ = 1,000,000, sliding window 256 on layers 0–9, a dense SwiGLU MLP on layer 0 and 24 routed experts of 192 with top-4 routing on layers 1–19, aux-loss-free bias balancing (bias_update_rate 3e-3), no shared expert, context 512, 32,768-entry vocabulary, untied embeddings. 397.7M parameters, 173.5M active per token. An independent re-implementation of the Sarvam 30B design; docs/ARCHITECTURE.md §10 says which of Sarvam's choices held up at this scale.

Use

python sample.py --ckpt <this folder> --prompt "हिन्दी साहित्य

"
python serve.py  --ckpt <this folder>        # continue-the-text page

To fine-tune it, finetune.py --ckpt <this folder> --qa pairs.jsonl takes question/answer JSONL; docs/DEVELOPING.md describes the format.

How to load

With transformers

The modelling code travels with the weights, so trust_remote_code=True is required and there is nothing to clone:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("toonist/AnuLM-Base-400M", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("toonist/AnuLM-Base-400M")

ids = tok("\u092d\u093e\u0930\u0924 \u0915\u0940 \u0930\u093e\u091c\u0927\u093e\u0928\u0940", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=False)[0]))

Greedy output is identical to the AnuLM repository's own generate, cached or not, and the tokenizer here agrees with bpe.py token for token; both are pinned by tests in that repository.

Load it in float32 — which the config now asks for, so the line above is enough. Do not force dtype=torch.bfloat16: this is a mixture-of-experts model whose router keeps a per-expert bias, and rounding that bias to 16 bits changes which experts fire. The output does not get slightly worse, it collapses into repeated tokens. The big tensors are stored in bfloat16 and upcast on load; the router bias and the norms are stored in float32 for this reason. For speed, use torch.autocast over float32 weights, which is how every number in this card was measured. The model loads the code from this repo at trust_remote_code=True; pin a revision if you want that fixed.

Batches must be unpadded (one sequence at a time), and beam search is not supported.

With the AnuLM repository

Every script there takes this folder wherever it takes a .pt:

hf download toonist/AnuLM-Base-400M --local-dir AnuLM-Base-400M
python serve.py  --ckpt AnuLM-Base-400M          # web page at http://127.0.0.1:8000
python sample.py --ckpt AnuLM-Base-400M --prompt "\u092d\u093e\u0930\u0924 \u0915\u0940 \u0930\u093e\u091c\u0927\u093e\u0928\u0940"

from model import load_checkpoint, AnuLM
ck = load_checkpoint("AnuLM-Base-400M")
m = AnuLM(ck["cfg"]).eval(); m.load_state_dict(ck["model"])

Files: model.safetensors (397.7M parameters), tokenizer.multi32k.json for bpe.BPE.load, and tokenizer.json in the tokenizers format for AutoTokenizer.