Gemma-3-1B-Code-Alpaca-FT

Gemma-3-1B-Code-Alpaca-FT

This is a fine-tuned version of the unsloth/gemma-3-1b-it model. It was fine-tuned by prathmesh-nik on the iamtarun/code_instructions_120k_alpaca dataset to enhance its ability to follow instructions related to code generation and general programming tasks.

Model Details

Technical Specifications

• Context Window: The model's configuration (config.json) specifies a theoretical maximum context of 32,768 tokens. However, it is based on unsloth/gemma-3-1b-it, which is optimized for an 8192 token context. Furthermore, this fine-tune was performed with a max_seq_length of only 2048 tokens. For best results, it is recommended to use this model with sequences close to the 2048 token training length.
• Max New Tokens: This is a parameter set during inference, not an intrinsic property of the model. The example code uses a default, but you can set this value higher or lower based on your needs (up to the context window limit).

Intended Use

This model is designed for general-purpose instruction following, with a focus on coding assistance. It can be used for tasks such as:

• Generating code snippets in various programming languages.
• Answering questions about programming concepts.
• Acting as a backend for a development-focused chatbot.

How to Use

You can use this model with the transformers library just like the base Gemma model.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Make sure you have logged in to huggingface-cli
# huggingface-cli login

model_id = "prathmesh_nik/Gemma-3-1B-Code-Alpaca-FT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

chat = [
    { "role": "user", "content": "Write a Python function to calculate the factorial of a number." },
]

prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)

outputs = model.generate(input_ids=inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Limitations and Bias

This model inherits the limitations and biases of the base unsloth/gemma-3-1b-it model and the training dataset.

• Short Context Training: Since the model was fine-tuned on sequences of only 2048 tokens, its performance on tasks requiring a very long context (i.e., approaching the 8192 token limit) may be degraded.
• Quantization-Aware Training: The adapters were trained on a 4-bit model. While the final model is full-precision, this process can introduce subtle differences compared to training in full precision.
• General Limitations: As a model fine-tuned on a specific dataset, it may not be as effective for general-purpose tasks and may not always provide correct answers. It may generate incorrect or insecure code and should not be used for mission-critical applications without human oversight.

Fine-tuning Details

Training Data

This model was fine-tuned on the iamtarun/code_instructions_120k_alpaca dataset, which contains a large set of instruction-response pairs for coding tasks.

Training Procedure

The model was fine-tuned using the LoRA (Low-Rank Adaptation) methodology. Key aspects of the training include:

• Unsloth Optimization: The training was performed using the Unsloth library, which enables significantly faster training and lower memory usage.
• 4-bit Quantization: For maximum efficiency, the LoRA adapters were trained on a 4-bit quantized version of the base model. The final merged model is in full precision (bfloat16).
• Parameter-Efficient Fine-Tuning (PEFT): Only a small fraction of the model's parameters (the LoRA adapters) were trained, making the process highly efficient.

Repository and Training Code

This model was fine-tuned by prathmesh_nik. The complete source code for the fine-tuning process, including data preparation, training, and merging scripts, is available on GitHub.

• GitHub Repository: [https://github.com/prathmeshnik]

Original Model Citation

Please cite the original Gemma 3 report if you use this model in your research:

@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}