Krix1-0.2b

Krix1-0.2b

A ~122M-parameter (≈0.2B) Llama-style decoder, pretrained from scratch on Hindi text. Every weight was trained from random initialization — this is not a fine-tune of an existing model.

What it is

• Architecture: Llama-style decoder — RMSNorm (pre-norm), RoPE, SwiGLU MLP, no biases, tied embeddings.
• Size: 12 layers, hidden 768, 12 heads, context 1024, vocab 48,000 → 121.8M params.
• Tokenizer: custom SentencePiece BPE (48k) trained on Hindi, with byte fallback.
• Data: AI4Bharat Sangraha (Hindi, Devanagari — verified + unverified), ~3B tokens seen during training out of ~19.6B available.
• Training: bf16, AdamW, cosine LR 6e-4→6e-5, ~0.5M tokens/step, ~6k steps on a single A100. Final val perplexity ≈ 19.

Honest limitations

This is a base completion model and a small one:

• It produces fluent, grammatical Hindi — that's what it's good at.
• It has no reliable world knowledge and will confidently make up facts. Knowledge needs far more scale.
• It is not instruction-tuned — it continues a prompt, it does not follow commands or chat.
• It only knows Hindi.

Usage

Tokenize with sentencepiece (the bundled hin_sp_48k.model) and run the model with transformers:

import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM

repo = "jokernifty/Krix1-0.2b"
sp = spm.SentencePieceProcessor(model_file=hf_hub_download(repo, "hin_sp_48k.model"))
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "भारत एक विशाल देश"
ids = torch.tensor([sp.encode(prompt)])
out = model.generate(ids, max_new_tokens=150, do_sample=True,
                     temperature=0.8, top_k=200, eos_token_id=2, pad_token_id=3)
print(sp.decode(out[0].tolist()))

Note: tokenization uses SentencePiece directly (not AutoTokenizer), because the tokenizer is a SentencePiece BPE model with byte fallback. This guarantees encoding identical to training.

License & data

Released for research and educational use. Trained on AI4Bharat Sangraha — review that dataset's terms before redistributing derivatives.