vrs-sft-Qwen2_5-7B-it

Qwen2.5-7B-Instruct-GGUF-SFT

Model Description

• Developed by: VRS Foundation
• Model type: Causal Language Model (Fine-tuned)
• Language(s): English
• Finetuned from model: Qwen/Qwen2.5-7B-Instruct

This model is a fine-tuned version of Qwen2.5-7B-Instruct optimized for conversational chat using the HH-RLHF dataset. It has been fine-tuned using LoRA and exported to the GGUF q4_k_m format for efficient inference on CPU and consumer-grade GPUs using llama.cpp or llama-cpp-python.

The fine-tuning process utilized the Unsloth library to significantly accelerate training while minimizing memory consumption.

Training Details

Dataset

The model was fine-tuned on a 1,000-sample subset of the Anthropic/hh-rlhf dataset to align it with helpful and harmless conversational patterns. The training focused on the chosen conversations ("chosen" split) formatted with Human and Assistant roles.

Hyperparameters and Configuration

• Max Sequence Length: 2048 tokens
• LoRA Rank (r): 16
• LoRA Alpha: 16
• LoRA Dropout: 0
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Batch Size: 2 (with 4 gradient accumulation steps, effective batch size = 8)
• Learning Rate: 2e-4
• Optimizer: adamw_8bit
• Max Steps: 60
• Warmup Steps: 5
• Weight Decay: 0.01
• Quantization: 4-bit (during training), exported to Q4_K_M GGUF

Usage

This model is packaged as a .gguf file (qwen2.5-7b-instruct.Q4_K_M.gguf), which can be loaded directly into UI frameworks like Streamlit using llama-cpp-python.

Setup Instructions

1. Install the required dependencies:

pip install llama-cpp-python streamlit

1. Run inference using Python and llama-cpp-python:

from llama_cpp import Llama

# Load the model (moves layers to GPU if n_gpu_layers=-1 and CUDA is available)
llm = Llama(
    model_path="qwen2.5-7b-instruct.Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=-1 
)

# Format the prompt using standard HH-RLHF format
prompt = "Explain the theory of relativity in simple terms."
formatted_prompt = f"Human: {prompt} \n\nAssistant:"

# Generate response
output = llm(
    formatted_prompt,
    max_tokens=256,
    temperature=0.7,
    stop=["Human:", "\n\n"],
    echo=False
)

print(output["choices"][0]["text"].strip())

Streamlit App Integration

This model was paired with a minimal Streamlit app (app.py) featuring an interactive chat UI. You can run it via:

streamlit run app.py

Prompt Format

For optimal performance and to maintain consistency with the fine-tuning process, use the following prompt format:

Human: {user_input} 

Assistant: