pothana-sp-base-300M

Pothana Base 300M

A 387M parameter LLaMA-style language model trained from scratch on Telugu text.

Named after Bammera Pothana, the celebrated 15th-century Telugu poet who authored the Andhra Maha Bhagavatamu.

Developed by Dvitva AI.

Model Details

Quick Start

Using pipeline

from transformers import pipeline

pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M", trust_remote_code=True)
result = pipe("తెలుగు భాష", max_new_tokens=50, do_sample=True, temperature=0.8)
print(result[0]["generated_text"])

Note: trust_remote_code=True is required for the custom tokenizer that cleans up SentencePiece word boundary markers for readable output.

Manual loading

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("dvitvaai/pothana-base-300M")
tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M", trust_remote_code=True)

text = "తెలుగు భాష చాలా అందమైనది"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Tokenizer

This model uses a SentencePiece Unigram tokenizer with a 48K vocabulary, trained directly on Telugu text.

• Handles raw Telugu text directly (no preprocessing needed)
• Byte-fallback for out-of-vocabulary characters
• Split digits for better number handling
• NFKC normalization

Architecture

Key features:

• Grouped Query Attention (GQA): 16 query heads, 4 KV heads — 4x KV cache reduction
• Block-wise Weight Sharing: 30 unique blocks, each used twice = 60 effective layers (MobileLLM-LS)
• SwiGLU MLP with 2816 intermediate size
• RoPE positional encoding (theta=10000.0)
• RMSNorm (no bias in any linear layer)

Training

• Data: Telugu text corpus (Sangraha dataset)
• Preprocessing: SentencePiece tokenization (raw text)
• Optimizer: AdamW (lr=3e-4, weight_decay=0.1, beta1=0.9, beta2=0.95)
• Schedule: WSD (Warmup-Stable-Decay)
• Precision: bf16 mixed precision
• Hardware: Single NVIDIA B200 GPU

Limitations

• This is a base model (not instruction-tuned) — it performs text completion, not instruction following
• Trained primarily on Telugu text; limited multilingual capability
• Small model size (387M) limits reasoning and knowledge capacity

License

Apache 2.0

Citation

If you use this model, please cite:

@misc{pothana-base-300M,
  title={Pothana Base 300M: A Telugu Language Model},
  author={Dvitva AI},
  year={2025},
  url={https://huggingface.co/dvitvaai/pothana-base-300M}
}