Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged

KhushalM/Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged

A causal language model fine-tuned to generate conceptual explanations in the style of Richard Feynman, combining supervised warm-up with on-policy reinforcement learning (Group Relative Proximal Optimization, GRPO) and accelerated inference via vLLM. The model excels at breaking down complex ideas into first-principles intuitions, concrete analogies, and short, clear paragraphs.

---

Model Details

Model Description

This model builds on Qwen/Qwen2.5-1.5B-Instruct, with a multi-stage training pipeline:

1. Supervised Fine-Tuning (SFT)

   • ~750 carefully designed prompt–completion pairs.
   • Each pair follows a Feynman-style teaching pattern:
     • simple language,
     • layered explanation,
     • everyday analogies,
     • checks for understanding.
   • Implemented with LoRA adapters for efficient parameter updates.

2. Reinforcement Learning with GRPO

   • Reward Model: trained on GPT-4o-mini preference judgments between candidate completions.
   • Objective: maximize adherence to “first-principle reasoning” and clarity of explanation.
   • Algorithm: Group Relative Proximal Optimization (GRPO), a variant of PPO adapted for group-based baselines.
   • On-policy rollouts were generated using the SFT-warm model, with iterative policy updates guided by the reward model.

3. Merging and Optimization

   • LoRA adapters from SFT and RLHF stages were merged into the base model.
   • Final model optimized for inference with vLLM, reducing latency and improving throughput.

• Finetuned by: Khushal Mandavia
• Model type: Decoder-only transformer (causal LM)
• Base model: Qwen/Qwen2.5-1.5B-Instruct
• Repository: huggingface.co/KhushalM/Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged

---

Uses

Direct Use

• Generate clear, analogy-driven explanations of technical or scientific topics.
• Serve as an educational chatbot that teaches concepts step-by-step.

Downstream Use

• Integrate into learning platforms for pedagogical clarity.
• Generate lecture notes, study guides, or FAQs.

Out-of-Scope Use

• Not intended for authoritative factual precision.
• Not optimized for open-domain chit-chat or casual conversation.

---

Bias, Risks, and Limitations

• Hallucinations: May invent plausible but incorrect details.
• Oversimplification: Risk of leaving out nuance in pursuit of clarity.
• Style Bias: Strongly favors the “Feynman teaching style,” which may be unsuitable for formal or technical contexts.

Recommendation: Fact-check outputs before use in professional or educational materials.

---

Training Procedure

Supervised Fine-Tuning

• Dataset size: ~750 curated examples.
• Framework: Hugging Face transformers with PEFT (LoRA).
• Loss: Cross-entropy, optimized with AdamW.

Reward Model

• Base: Qwen2.5-1.5B classifier head.
• Training Data: ~3k pairwise preferences (generated via GPT-4o-mini comparisons).
• Objective: Predict preference for completions that are simpler, analogy-rich, and structured.

GRPO (Reinforcement Learning Phase)

• Algorithm: Group Relative Proximal Optimization (variant of PPO).
• Rollouts: Generated with SFT-warm policy.
• Group Baseline: Used relative ranking of completions within a rollout batch to stabilize training.
• Reward: Scalar from reward model (0–1).
• Stability tricks: Gradient clipping, KL penalty, early stopping.

Merge + Inference

• LoRA adapters from SFT and GRPO merged.
• Final model loaded into vLLM for inference optimization.
• Latency reduced by ~2.5× compared to vanilla transformers inference.

---

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "KhushalM/Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", trust_remote_code=True)

generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device="cuda"  # or "cpu"
)

prompt = "Explain quantum entanglement from first principles."
output = generator(prompt, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(output[0]["generated_text"])