Downloads · 30 days
0
qwen2.5-1.5b-amid-nnm
VoCuc/qwen2.5-1.5b-amid-nnm
qwen2.5-1.5b-amid-nnm is a text generation model from VoCuc. Use it when you need the model to write or continue text. It is set up for peft.
Student distilled from Qwen2.5-14B-Instruct (teacher) into Qwen2.5-1.5B-Instruct (student) with the AMiD (alpha-mixture assistant distribution) knowledge-distillation framework.
Access
Public
Updated Jul 11, 2026
Repo size
516 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin504 MB · 97%
At a glance
- Task
- Text Generation
- Library
- peft
- Model type
- qwen2
- Access
- Public
- Created
- Jul 11, 2026
- Updated
- Jul 11, 2026
- SHA
- f0d9b03f
Try a prompt
Base models
Model card
From the Hugging Face model README
AMiD - distillm-NNM (SFKL + NNM, frozen projectors)
Student distilled from Qwen2.5-14B-Instruct (teacher) into Qwen2.5-1.5B-Instruct (student) with the AMiD (alpha-mixture assistant distribution) knowledge-distillation framework.
Method
Adaptive SFKL (skew-forward KL, skew_alpha=0.1) + NNM (nearest-neighbour mixture; K=128, 4 layers, d'=256) with frozen projectors (no_train_proj). On-policy student-gen with adaptive threshold + replay buffer (capacity 1000).
Training
| Teacher | Qwen/Qwen2.5-14B-Instruct (14.8B) |
| Student base | Qwen/Qwen2.5-1.5B-Instruct (1.5B) |
| Data | UltraInteract - 79,751 prompts (200 held out for dev) |
| Epochs / optimizer steps | 2 / 4984 |
| LR / schedule | 1e-4, cosine, weight decay 1e-2, grad clip 1.0 |
| Batch | 8 x grad-accum 2 x 2 GPUs |
| LoRA | r=16, alpha=64, dropout=0.05 |
| Precision | bf16 student (DeepSpeed ZeRO-1), fp16 teacher |
| Hardware | 4x NVIDIA H200 NVL |
| Final dev ROUGE-L | 7.4443 |
Usage - LoRA adapter (always available, in lora_adapter/)
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-1.5B-Instruct", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "VoCuc/qwen2.5-1.5b-amid-nnm", subfolder="lora_adapter")
model = model.merge_and_unload() # <- reconstructs the merged model exactly
tok = AutoTokenizer.from_pretrained("VoCuc/qwen2.5-1.5b-amid-nnm")
Usage - merged model (if the root model.safetensors is present)
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("VoCuc/qwen2.5-1.5b-amid-nnm", dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("VoCuc/qwen2.5-1.5b-amid-nnm")
The merged weights are exactly Qwen2.5-1.5B-Instruct + this LoRA adapter, so the two paths above
are equivalent. training_args.json and training_log.txt hold the full config and training log.
- Task
- Text Generation
- Library
- peft
- Type
- qwen2
- Created
- Jul 11, 2026
- Updated
- Jul 11, 2026