Dragon-1-0.5B

Dragon-1-0.5B (qwen2-0.5b-reasoning v2.0.1)

Dragon is a lightweight code reasoning and generation model built upon the base Qwen2-0.5B-instruct model. It offers accurate and quick code snippet and long-form code generation in all major programming languages. It's small size (0.5B parameters) allows it to run comfortably on most laptop/commercial grade GPUs. This model also offers Q/A and subject matter expert capabilities on code related subjects.

The Dragon-1 is the pilot model for the Dragon-1/1.5 series which incorporates high-end reasoning capabilities into the standard Qwen2 and Qwen2.5 architectures.

The 0.5B variant has been SFT trained on code reasoning traces found here with further RL training carried out via. a GRPO algorithm. This endows the model with enhanced reasoning capabilities which allows it to serve higher quality and hallucination-free generations.

---

Estimated parameters: ~0.5B

Architecture: Qwen2

Intended use: Code snippet and long-form generations from natural language, instruction following and advanced reasoning

---

Training data

Phase-1

• Source: deepseek-v4-reasoning-code-2500 dataset (https://huggingface.co/datasets/Banaxi-Tech/Deepseek-V4-Reasoning-Code-2500)
• Rows: ~2,555 rows templated with a custom .jinja chat format
• Training: trained for 5,000 steps on an RTX PRO 4000 (24GB VRAM)

Phase-2

• Source: deepseek-v4-reasoning-code-2500 dataset (https://huggingface.co/datasets/Banaxi-Tech/Deepseek-V4-Reasoning-Code-2500)
• Rows: ~2,555 rows templated with a custom .jinja chat format
• Training: trained via. GRPO for 500 steps on an RTX PRO 4000 (24GB VRAM)

Usage

Install requirements:

pip install -r requirements.txt
pip install transformers datasets accelerate safetensors

Usage (Hugging Face Hub)

You can load it directly from HuggingFace:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer


device = "cuda" if torch.cuda.is_available() else "cpu"

model_id = "DireDreadlord/Dragon-1-0.5B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="auto"
)
model.to(device)
tokenizer = AutoTokenizer.from_pretrained(model_id)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)

prompt = "Can you reason about the following leetcode question: Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. Please provide a detailed reasoning and explanation for your answer."

input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    return_tensors="pt",
    tokenize=True,
)["input_ids"].to(device)

output = model.generate(
    input_ids,
    do_sample=True,
    temperature=0.4,
    top_k=50,
    repetition_penalty=1.05,
    max_new_tokens=2048,
    streamer=streamer,
)

For optimal long-form generation(with reasoning), set max_new_tokens=2048

Limitations

• Model for experimental use only; users should employ it as such under license.