Kemcho-Gemma-3-1B-IT

Kemcho-Gemma-3-1B-IT

Kemcho-Gemma-3-1B-IT is a fine-tuned model from google/gemma-3-1b-it for Gujarati instruction following and general assistant tasks. The training used LoRA and was merged into the base weights for single-repo deployment.

• Architecture: Gemma 3 (1B)
• Weights: model.safetensors (~4 GB, bf16 recommended)
• Data: Alpaca-Gujarati (~10,000 example subset)
• Best for: Gujarati chat, rewriting, summarization, simple Q&A

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "shalinm/Kemcho-Gemma-3-1B-IT" 

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "તમે મદદરૂપ ગુજરાતી સહાયક છો."},
    {"role": "user", "content": "આપેલ વાક્યોને એક વાક્યમાં ભેગા કરો.\n\nહું કામ માટે મોડો પડ્યો હતો. મારે મારા બાળકોને શાળાએ લઈ જવાનું હતું."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Details

• Base model: google/gemma-3-1b-it
• Training method: SFT using LoRA, then merged into base weights
• LoRA config (pre-merge):
  • rank: 16
  • alpha: 32
  • dropout: 0.1
  • target modules: attention q_proj, k_proj, v_proj, o_proj, mlp up_proj, down_proj,gate_proj
• Trainable params (pre-merge): ~9.69M of ~1.01B (~0.96%)
• Precision: trained/evaluated in bfloat16

---

Training

• Hardware: 1 x A40
• Epochs: 3
• Logged steps: 300

Training/Validation Loss (snapshot)

---

Intended Uses and Limitations

• Intended: Gujarati assistant tasks—Q&A, rewriting, summarization, everyday instructions.
• Not intended: Safety-critical uses, factual lookup without verification, generating harmful content.

Limitations

• May hallucinate facts or reflect biases in data.
• Non-Gujarati performance is not specifically tuned.

Safety

• Use with content filtering and human oversight for sensitive domains.
• Consider additional alignment or safety fine-tuning for production.

---

License

• Weights derived from google/gemma-3-1b-it. Use is subject to the Gemma license and terms set by Google.
• Ensure compliance with the licenses of any datasets used during fine-tuning.