ml-hamz-colab1-grpo_unsloth

Uploaded model

• Developed by: tradexy
• License: apache-2.0
• Finetuned from model : unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit

This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>

ml-hamz-colab1-grpo_unsloth

Model Overview

This model is a fine-tuned version of the Qwen2.5-3B-Instruct architecture, optimized for instruction-following tasks. The fine-tuning process was conducted using the GRPO (Gradient Reversal Pretraining Optimization) method to enhance performance on specific queries.

Training Details

The fine-tuning was performed using the Unsloth Qwen2.5 (3B) GRPO Notebook. Key aspects of the training include:

• Dataset: Custom dataset focusing on instruction-based queries.
• Training Duration: Approximately 2 hours on a single GPU (T4 GPU - colab)
• Challenges: Managed limited disk space by deleting intermediate files during the process.
• Observation During Training: Reward increased as model answered questions correctly and completion_length became shorter.

Step - Training Loss - reward - reward_std - completion_length - kl

1 - 0.000000 - 0.125000 - 0.000000 - 200.000000 - 0.000000

2 - 0.000000 - 0.072375 - 0.248112 - 200.000000 - 0.000000

3 - 0.000000 - -0.079000 - 0.163776 - 182.500000 - 0.000005

Usage Instructions

To utilize this model for inference:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tradexy/ml-hamz-colab1-grpo_unsloth")
model = AutoModelForCausalLM.from_pretrained("tradexy/ml-hamz-colab1-grpo_unsloth")

input_text = "Your instruction here"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))


For local deployment on Windows (can also use wsl ubuntu) see Machine Learning With Hamza
https://github.com/Hmzbo/Fine-tune-LLMS-with-grp