qwen2.5-coder-3b-instruct-awq-gguf

Qwen2.5-Coder-3B Distilled Model

This is a knowledge-distilled version of Qwen2.5-Coder-3B-Instruct-AWQ, trained using knowledge distillation from Qwen2.5-Coder-7B-Instruct-AWQ.

Model Details

• Base Model: Qwen/Qwen2.5-Coder-3B-Instruct-AWQ
• Teacher Model: Qwen/Qwen2.5-Coder-7B-Instruct-AWQ
• Training Method: Knowledge Distillation with LoRA
• Best Validation Loss: 1.9286
• Training Time: ~5 minutes
• Parameters Trained: 14.9M (4.59% of base model)

Training Configuration

• Temperature: 2.0 (optimal)
• Alpha: 0.95 (95% distillation weight)
• LoRA Rank: 8
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-3B-Instruct-AWQ",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-3B-Instruct-AWQ")

# Load distilled adapter
model = PeftModel.from_pretrained(base_model, "Vinitha2004/qwen2.5-coder-3b-instruct-awq-gguf")

# Generate code
input_text = "Original Code:\ndef add(a, b):\n    return a + b\n\nUpdate Snippet:\n// ... existing code ...\ndef add(a: int, b: int) -> int:\n// ... existing code ...\n\nUpdated Code:\n"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

Performance

This distilled model retains the knowledge from the 7B teacher model while being significantly more efficient:

• Faster inference (3B vs 7B parameters)
• Lower memory usage
• Maintained code generation quality

Training Dataset

Trained on 5000 code editing examples from custom dataset.

Files

• adapter_config.json: LoRA configuration
• adapter_model.safetensors: Trained LoRA weights (59MB)
• Other standard tokenizer files