mimo-7b-gdn-hybrid-init

mimo-7b-gdn-hybrid-init

Surgery/init checkpoint — uniform 1:4 GDN conversion of MiMo-7B-RL-0530 with plain weight inheritance, before any training. vLLM parity at init: 5.3x teacher floor (the standing regression metric re-measured at each gate).

Part of the MiMo second-model arm of a study converting full-attention models into GDN (gated DeltaNet) hybrids (uniform 1:4 retention) and recovering capability via staged distillation. Teacher/base: MiMo-7B-RL-0530.

NOTE: these checkpoints use the custom mimo_gdn architecture — register it before loading (import src.models.mimo_gdn from the project repo, or use the load_causal_lm helper in src/eval/sample_generations.py). Bare AutoModelForCausalLM will not work. vLLM serving needs src.models.vllm_mimo_gdn.register().

Pipeline: surgery/init (uniform 1:4 plain inheritance) -> stage-1 hidden-state alignment -> stage-2a forward-KL KD @4k -> stage-2b long-context KD @32k. Within-arm reference: teacher wikitext-2 PPL 27.51 (RL'd teacher; never cross-arm-compare absolute PPL).