gemma-3-1b-it-pms-32768

gemma-3-1b-it-pms-32768

This model is a 26.43% smaller version of google/gemma-3-1b-it optimized for Piedmontese language via vocabulary size reduction using the trimming method.

This trimmed model should perform similarly to the original model with only 32,768 tokens and a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in Piedmontese were removed from the vocabulary.

Model Statistics

Mining Dataset Statistics

• Number of texts used for mining: 200,000 texts
• Dataset: Lumberjackk/fineweb-2-trimming

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "provisoirement/gemma-3-1b-it-pms-32768"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

prompt = "Your prompt in Piedmontese."
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(**model_inputs, max_new_tokens=256)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]
response = tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)

Citation

Gemma 3

@misc{gemmateam2025gemma3technicalreport,
      title={Gemma 3 Technical Report},
      author={Gemma Team},
      year={2025},
      eprint={2503.19786},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.19786},
}