qwen2.5-1.5b-amid-feature

AMiD - feature variant (AB-divergence + hidden-state feature MSE)

Student distilled from Qwen2.5-14B-Instruct (teacher) into Qwen2.5-1.5B-Instruct (student) with the AMiD (alpha-mixture assistant distribution) knowledge-distillation framework.

Method

AMiD AB-divergence (div=ab, order=pr, alpha=0.5, lambda=0.5) plus a hidden-state feature-matching MSE: student layers [8,13,18,23] are mapped through learned linear projectors (1536 -> 5120) onto teacher layers [14,22,31,40]. On-policy student-gen with replay buffer.

Reproducibility note: the feature-loss loop in f_base_finetune.py originally passed the layer-index lists into the projector instead of the hidden-state tensors at those layers (TypeError: linear(): input must be Tensor, not int). This run applied the fix: index the captured hidden states (s_hidden[s_idx], teacher_outputs.hidden_states[t_idx]) and compute the MSE in fp32.

Training

Usage - LoRA adapter (always available, in lora_adapter/)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "VoCuc/qwen2.5-1.5b-amid-feature", subfolder="lora_adapter")
model = model.merge_and_unload()          # <- reconstructs the merged model exactly
tok = AutoTokenizer.from_pretrained("VoCuc/qwen2.5-1.5b-amid-feature")

Usage - merged model (if the root model.safetensors is present)

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("VoCuc/qwen2.5-1.5b-amid-feature", dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("VoCuc/qwen2.5-1.5b-amid-feature")

The merged weights are exactly Qwen2.5-1.5B-Instruct + this LoRA adapter, so the two paths above are equivalent. training_args.json and training_log.txt hold the full config and training log.