gemma3-270m-nepali

Gemma 3 270M — Nepali SFT (LoRA Adapter)

A LoRA adapter for google/gemma-3-270m-it (~270M parameters), instruction fine-tuned on Nepali conversational data. Trained with Unsloth starting from the pre-quantized checkpoint unsloth/gemma-3-270m-it-unsloth-bnb-4bit on a free Google Colab T4 GPU.

Model Details

Intended Uses

This adapter adapts Gemma 3 270M to follow instructions and hold conversations in Nepali. It's aimed at:

• Lightweight Nepali instruction-following / chat experimentation
• Low-resource / edge deployment scenarios where a ~0.3B model is required
• A baseline to compare against larger Nepali fine-tunes

Out of scope: factual accuracy on specialized domains (medical, legal, financial), tasks requiring long-context reasoning, and any high-stakes or safety-critical use — this is a small model fine-tuned for a modest number of steps and has not been evaluated for those purposes.

Training Data

• Dataset: himalaya-ai/nepali-sft-compile
• Format: multi-turn conversations field ({"from": "human"|"gpt", "value": "..."}), converted to Gemma 3's chat template (<start_of_turn> / <end_of_turn>) via tokenizer.apply_chat_template
• Rows used: first 30,000 rows of the dataset
• Split: 90% train / 10% validation (seed 3407)

Training Procedure

LoRA Configuration

Hyperparameters

How to Use

  from unsloth import FastLanguageModel

 model, tokenizer = FastLanguageModel.from_pretrained(
  model_name = "unsloth/gemma-3-270m-it-unsloth-bnb-4bit",
  max_seq_length = 2048,
  dtype = None,
  load_in_4bit = True,
)
# Load this adapter on top of the base model
model.load_adapter("<path-or-repo-of-this-adapter>")
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "नृत्य प्रतियोगिताहरूले नर्तकहरूको विकासमा पार्ने प्रभावबारे चर्चा गर्नुहोस्।"}]
inputs = tokenizer.apply_chat_template(
  messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(input_ids=inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True)
print(tokenizer.decode(output[0][inputs.shape[-1]:],
skip_special_tokens=True))

Or with plain 🤗 Transformers + PEFT:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
  import torch

  base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-270m-it", torch_dtype=torch.float16, device_map="auto"
  )
model = PeftModel.from_pretrained(base, "<path-or-repo-of-this-adapter>")
tokenizer = AutoTokenizer.from_pretrained("<path-or-repo-of-this-adapter>")

Limitations & Biases

• Trained on a subset (30k rows) of a single compiled dataset; coverage of topics, dialects, and registers of Nepali reflects that dataset's composition and may not generalize broadly.
• As a ~270M-parameter model, it has limited reasoning and world-knowledge capacity compared to larger LLMs — expect shorter-context tasks and simpler instructions to work best.
• Not evaluated against a held-out benchmark or human preference study; validation loss was used to monitor training but does not guarantee output quality or safety.
• Inherits any limitations and biases present in the base google/gemma-3-270m-it model and in the source dataset.

Framework Versions

• Unsloth (latest at time of training)
• TRL ≥ 0.12.0
• PEFT
• Transformers (version pinned by Unsloth's dependency resolution)
• bitsandbytes