kasturi-500m

Kasturi-500M

Status: 🟢 PRETRAINING IN PROGRESS — launched 2026-06-29 07:03 UTC. ETA: ~2026-07-24 (3.5 weeks, 30B tokens, 57,220 optimizer steps). Single 16 GB consumer GPU (NVIDIA RTX 5060 Ti).

Kasturi (ಕಸ್ತೂರಿ) is an open-source bilingual Kannada + English language model, trained from scratch on a single consumer GPU for the Karnataka developer community.

Quick facts

What's planned for launch

• 🪶 Native Kannada-first behavior (no English bias baked in)
• 🔄 Built-in KN ↔ EN translation
• 🧠 Tool-calling fallback when the model doesn't know
• 📦 GGUF release for Ollama / llama.cpp out of the box
• 📝 Markdown by default; JSON / HTML / plain-text overrides
• 🎯 Designed for offline use on consumer hardware

Why bilingual?

Most generalists ship with <1% Kannada (Llama, Gemma); even Sarvam-2B is ~10%. Kasturi flips the ratio. A model that thinks natively in Kannada — not one that translates from English internally — is what the Karnataka community has been missing.

What's NOT here yet

• Model weights — pre-training in progress
• Full technical model card
• Benchmark numbers
• Inference examples
• Tokenizer file

All of the above will land when the model finishes training. Watch this space.

Status updates

Follow progress on the project's working repository (link coming soon).

Citation

@misc{kasturi-2026,
  title     = {Kasturi: A Bilingual Kannada-English LLM Trained from Scratch on a Consumer GPU},
  author    = {Anand Kaman},
  year      = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/anandkaman/kasturi-500m}}
}

Training progress

Phase 3 schedule

• Steps 0 → 2,000 — warmup (LR 0 → 3e-4)
• Steps 2,000 → 51,498 — stable (constant LR 3e-4, main data mix: KN 65% / EN 30% / codemix 5%)
• Steps 51,498 → 57,220 — annealing (LR linear → 0, anneal mix: KN 50 / EN 20 / codemix 5 / math 12 / instruction 8 / code 5)
• Effective batch: 524,288 tokens/step (bf16 mixed-precision, gradient checkpointing, intra-document attention masking)
• Checkpoints: weight-only safetensors every 1K steps, full resume every 2.5K steps

Architecture (locked, see model files for full spec)

• 538.7M parameters · Llama-style decoder-only
• 32 layers · d_model 1024 · d_ff 3584 (3.5× ratio) · GQA 16/4
• 4K native context · RoPE θ=500K · Dynamic-YaRN inference extension to 8K
• 100K SentencePiece BPE vocab · KN fertility 2.09 (beats IndicSuperTokenizer 2.19)
• RMSNorm + RoPE + SwiGLU + tied embeddings + intra-doc attention mask

License

Apache 2.0 — see LICENSE once released.

---

Built solo, from scratch, for Karnataka. By @anandkaman.