Menda-3B-500

Menda-3B-500: GRPO-Tuned Qwen2.5 Model

Menda-3B-500 is a fine-tuned version of Qwen2.5-3B-Instruct, trained with GRPO (Guided Reinforcement from Preference Optimization) for 500 steps. This model shows improved performance on reasoning benchmarks compared to the base model.

Model Details

• Base Model: Qwen/Qwen2.5-3B-Instruct
• Training Method: GRPO (Guided Reinforcement from Preference Optimization)
• Training Steps: 500
• Parameters: 3 billion
• Context Length: 32K tokens
• Training Data: GSM8K (mathematical reasoning)
• Chat Template: Uses the Qwen2 chat template

Chat Format

This model uses the standard Qwen2 chat template. For best results when using the model directly, format your prompts as follows:

<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

When using the model through the Hugging Face Transformers library, the chat template will be applied automatically when using the chat_template functionality:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "weathermanj/Menda-3B-500"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain the concept of machine learning in simple terms."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=300)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Benchmark Results

Menda-3B-500 has been evaluated on several standard benchmarks:

MMLU Performance

Key Strengths

• Balanced Performance: Maintains strong performance across diverse tasks with minimal trade-offs.
• Improved BoolQ: Achieves 90% on BoolQ, showing excellent reading comprehension capabilities.
• Strong Reasoning: Maintains 90% on both PIQA and Winogrande, demonstrating robust reasoning abilities.
• Efficient Training: Achieves impressive results with relatively minimal training (500 steps).
• Stable Knowledge: Maintains strong MMLU performance (68.60%) across diverse knowledge domains.

Usage Examples

Basic Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "weathermanj/Menda-3B-500"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

prompt = "Explain the concept of machine learning in simple terms."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=300)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Chat Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "weathermanj/Menda-3B-500"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Give me a short introduction to large language models."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Using with Ollama

You can also use this model with Ollama by converting it to GGUF format:

# Convert to GGUF
python -m llama_cpp.convert_hf_to_gguf weathermanj/Menda-3B-500 --outfile menda-3b-500.gguf

# Create Ollama model
cat > Modelfile << EOF
FROM menda-3b-500.gguf
TEMPLATE """{{ .Prompt }}"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER top_k 40
EOF

ollama create menda-3b-500 -f Modelfile
ollama run menda-3b-500

Training Configuration

The model was trained using the GRPO methodology with the following configuration:

• LoRA Rank: 128
• Learning Rate: 5e-6
• Optimizer: AdamW (8-bit)
• Batch Size: 8 per device
• Gradient Accumulation Steps: 4
• Training Samples: 100 examples from GSM8K

License

This model inherits the license of the base Qwen2.5-3B-Instruct model. Please refer to the Qwen2 license for details.