kannada-gemma-4b

ಕನ್ನಡ Gemma 4 (Kannada Gemma 4)

A Kannada-language-adapted LLM built by continual pre-training (CPT) and supervised fine-tuning (SFT) on Google's Gemma 4 E4B model, following the SambaLingo methodology.

Model Details

Training Pipeline

┌─────────────────────────────────────────────────────────────┐
│  Stage 1: Continual Pre-Training (CPT)                      │
│  ─────────────────────────────────────────                  │
│  Data: ~36 GB Kannada text + English (3:1 KN:EN ratio)     │
│  Sources: Sangraha, CulturaX-Kn, C4-Kn, IndicCorpV2,       │
│           Wikipedia-Kn, kannada_corpus_1m                   │
│  Hyperparams: lr=1e-4, cosine, wd=0.1, 4 epochs,           │
│               seq_len=4096, packing=True, bf16              │
│  Full-parameter (NOT LoRA) per SambaLingo findings          │
├─────────────────────────────────────────────────────────────┤
│  Stage 2: Supervised Fine-Tuning (SFT)                      │
│  ─────────────────────────────────────                      │
│  Data: Kannada instruction datasets                         │
│  Sources: Kannada_Bilingual_Instruct, indic-align,          │
│           hh_dpo_kannada (chosen responses), Aya_Kannada     │
│  Hyperparams: lr=2e-5, linear, 3 epochs, seq_len=2048,     │
│               assistant_only_loss=True, bf16                 │
├─────────────────────────────────────────────────────────────┤
│  Stage 3: DPO (optional)                                    │
│  ─────────────────────                                       │
│  Data: Cognitive-Lab/hh_dpo_kannada_translated              │
│  Hyperparams: lr=5e-7, β=0.1, linear, 3 epochs              │
└─────────────────────────────────────────────────────────────┘

Datasets Used

CPT (Continual Pre-Training)

SFT (Instruction Tuning)

Hardware Requirements

How to Run on Your GPU

1. Install dependencies

pip install torch transformers trl accelerate datasets peft sentencepiece
# For accelerated attention (Ampere+ GPUs only: A100, RTX 3090/4090, etc.)
pip install kernels~=0.12.0

2. Prepare CPT data

# Streams all Kannada datasets, mixes with English at 3:1 ratio
# Uses streaming mode — works even with low RAM
python prepare_cpt_data.py --output_dir ./data/cpt_kannada

# Include transliterated Kannada (Latin script) for extra data
python prepare_cpt_data.py --include_latn --output_dir ./data/cpt_kannada

# Adjust English ratio (default 25% = 3:1 KN:EN)
python prepare_cpt_data.py --english_ratio 0.20 --output_dir ./data/cpt_kannada

3. Prepare SFT data

python prepare_sft_data.py --output_dir ./data/sft_kannada

4. Run CPT training

# Full-parameter CPT (recommended — needs A100 80GB)
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --per_device_train_batch_size 4 \
  --gradient_accumulation_steps 8 \
  --learning_rate 1e-4 \
  --num_epochs 4

# LoRA CPT (for 24GB GPUs — lower quality but works)
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --use_lora \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 16

# Resume from checkpoint
python train_cpt.py \
  --data_dir ./data/cpt_kannada \
  --output_dir ./checkpoints/cpt \
  --resume_from_checkpoint ./checkpoints/cpt/checkpoint-5000

5. Run SFT training

# Full-parameter SFT
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 4 \
  --learning_rate 2e-5 \
  --num_epochs 3

# LoRA SFT (for 24GB GPUs)
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --use_lora

6. (Optional) Run DPO

python train_dpo.py \
  --model_path ./checkpoints/sft \
  --dataset_id Cognitive-Lab/hh_dpo_kannada_translated \
  --output_dir ./checkpoints/dpo \
  --learning_rate 5e-7 \
  --beta 0.1 \
  --num_epochs 3

7. Push to HuggingFace Hub

# Add --push_to_hub --hub_model_id to any training command
python train_sft.py \
  --model_path ./checkpoints/cpt \
  --data_dir ./data/sft_kannada \
  --output_dir ./checkpoints/sft \
  --push_to_hub --hub_model_id harshaperla/kannada-gemma-4b-sft

8. Inference

# Interactive chat
python inference.py --model_path ./checkpoints/sft

# Evaluate on FLORES translation benchmark
python inference.py --model_path ./checkpoints/sft --eval_flores

# Generate from file
python inference.py --model_path ./checkpoints/sft --input_file prompts.txt

Key Design Decisions

Why not extend the tokenizer?

Gemma 4's 262K vocabulary already handles Kannada well. Our fertility test showed:

• Kannada: 0.27–0.41 tokens/char (vs English: 0.17–0.23)
• Token fertility ratio: ~1.7x English (vs 5–10x for Llama 2's 32K vocab)
• Multi-character Kannada subwords already exist (e.g., ಕನ್ನಡ is a single token)
• Conclusion: no tokenizer extension needed

Why full-parameter instead of LoRA?

The SambaLingo paper explicitly showed full-parameter CPT >> LoRA CPT across all 9 languages tested. LoRA is available as a fallback for GPUs with <80GB VRAM.

Why the 3:1 Kannada:English ratio?

SambaLingo found that mixing English data prevents catastrophic forgetting of the model's general capabilities while learning the new language. The 1:3 English:Kannada ratio was optimal.

Why SambaLingo methodology?

SambaLingo (arXiv 2404.05829) achieved SOTA results across 9 languages using this exact pipeline. It's the best-validated language adaptation methodology currently available.

References

• SambaLingo: Ristić-Arsova et al., "SambaLingo: Teaching Small Large Language Models New Languages", arXiv:2404.05829
• Gemma 4: Google, "Gemma 4 Technical Report"
• Sangraha: AI4Bharat, largest Indic language corpus
• IndicCorpV2: AI4Bharat, Indic language corpus
• CulturaX: USTC, multilingual cleaned corpus

Citation

@misc{kannada-gemma-4b,
  author = {harshaperla},
  title = {Kannada Gemma 4: A Kannada-Language Adapted LLM},
  year = {2026},
  url = {https://huggingface.co/harshaperla/kannada-gemma-4b},
}

<!-- ml-intern-provenance -->

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'harshaperla/kannada-gemma-4b'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.