qwen-coder-0.5b-kd-mbpp

Qwen2.5-Coder-0.5B distilled from a 32B teacher (Knowledge Distillation + LoRA)

  A LoRA adapter for   Qwen2.5-Coder-0.5B-Instruct  , trained with   hybrid knowledge distillation (KD)  from  Qwen2.5-Coder-32B-Instruct   on Python programming problems from MBPP. The whole pipeline — calling the teacher, training the student, and evaluating both — runs on an 8GB Apple M2 MacBook, because only a ~10-30MB adapter is trained locally; the 32B teacher runs remotely via the HuggingFace Inference API.  

What is knowledge distillation?

  Knowledge distillation transfers capability from a large, expensive "teacher" model into a small, cheap "student" model. Instead of training the student only on a fixed dataset of correct answers (hard labels), KD also lets the student see  how the teacher thinks : which tokens the teacher considered likely at each step, and how confident it was among the alternatives. That extra signal — the teacher's probability distribution over its vocabulary, not just its final answer — is typically far richer than a single "correct" label, and is why distilled students often generalize better than students trained on hard labels alone.   This project combines   two   distillation signals into one loss: Loss = (1 - α) · CE_loss  +  α · KL_loss  

•   CE_loss — Sequence-level KD (hard-label).   Standard next-token   cross-entropy, but the "ground truth" is the   teacher's own generated   solution   for each MBPP problem, not MBPP's human reference solution. The   student is directly imitating what the teacher actually wrote.
•   KL_loss — Soft-label KD.   At every generated token position, the teacher   API returns its top-5 most likely tokens with their log-probabilities. We   renormalize those 5 probabilities to sum to 1, gather the student's own   probabilities for those  same  5 token ids, renormalize those too, and   compute the KL divergence between the two distributions. This teaches the   student not just  what  the teacher wrote, but  how confident it was and   what alternatives it seriously considered  at each step.   Both terms are combined with a weight α (ALPHA) that balances "imitate the final text" against "imitate the token-level thinking."  

Why teacher and student must share an architecture family

  The soft-label (KL) trick above only works because   teacher and student are both Qwen2.5-Coder models   and therefore share an   identical tokenizer and vocabulary  . The teacher API returns its top-5 tokens as strings; those strings are re-encoded with the student's own tokenizer to recover vocabulary ids. Because both models were tokenized the same way, a token string from the teacher maps 1:1 onto a real id in the student's vocabulary — so the KL divergence is comparing probabilities over the  same  discrete outcomes for both models. If the teacher and student came from different model families (e.g. a Llama teacher distilling into a Qwen student), their tokenizers would segment text differently, token strings would frequently fail to map onto a single student token id, and the soft-label KL term would break down. Any position where the mapping fails is simply excluded from the KL loss (masked out) and falls back to relying on the CE loss alone for that token.  

What is LoRA, and why use it here

LoRA (Low-Rank Adaptation)  freezes all of the base model's pretrained weights and injects small trainable "adapter" matrices alongside a chosen subset of layers. Instead of updating a full weight matrix W (huge, and expensive to store gradients/optimizer state for), LoRA learns a low-rank update ΔW = B·A where A and B are much smaller matrices of rank r. Only A and B are trained; W never changes and never needs gradients or optimizer state computed for it.   This is what makes training a 0.5B model on 8GB of RAM (and here, on CPU) practical. Anyone can later attach this adapter on top of the public base model to reproduce the distilled behavior, without ever downloading a separately fine-tuned 0.5B checkpoint.  

Pipeline

  1 - Query the 32B teacher (via Inference API) on MBPP problems; save its generated solution + top-5 token logprobs at every position | 2 - Fine-tune the 0.5B student with LoRA using the hybrid KD loss above | 3 - Run both the base model and the distilled student on held-out MBPP test problems; execute generated code against MBPP's asserts | 4 - Push the LoRA adapter + this model card to the Hub |  

Hyperparameters

  Knowledge distillation 

• ALPHA -  0.5 | Weight of soft (KL) loss vs hard (CE) loss
• TEMPERATURE | 2.0 | Softens both distributions before KL; kept lower than the classic Hinton default of 4.0 because a top-5 distribution is already truncated, and over-softening a 5-way distribution destroys signal
• Teacher decoding | temperature=0.0 (greedy) | Deterministic — captures the teacher's single "best" answer per problem
• MAX_NEW_TOKENS | 512 | Cap on teacher solution length
• TOP_LOGPROBS | 5 | Alternatives saved per generated token     LoRA 
• r (rank) | 8 | Rank of the adapter matrices; controls adapter capacity
• lora_alpha | 16 | Scaling factor (kept at the standard 2x-rank heuristic)
• lora_dropout | 0.05 | Dropout on adapter activations during training
• target_modules | q_proj, k_proj, v_proj, o_proj | Attention projections — where most task adaptation happens   Training 
• MAX_LENGTH | 768 tokens (prompt + solution, truncated)
• BATCH_SIZE | 1 (per micro-step)
• GRAD_ACCUM | 8 (effective batch size = 8)
• EPOCHS | 3
• LR | 2e-4
• Hardware | 8GB Apple M2, CPU (fp32) |