Downloads · 30 days
10
17% of all-time downloads
arianraje/mimo-7b-gdn-hybrid-stage1-align
mimo-7b-gdn-hybrid-stage1-align is a text generation model from arianraje. Use it when you need the model to write or continue text. The card lists the license as mit.
Stage-1 endpoint — hidden-state alignment. Exit: rel-MSE 0.0077, wikitext PPL 52.9 (1.9x teacher). Accepted via KD-smoke functional gate (val KL 1.554-0.563 in 2 steps); raw-prose degeneracy at this stage is MiMo-spec…
Downloads · 30 days
10
17% of all-time downloads
All-time downloads
58
Public
Parameters
8.3B
16.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.6 GB · 100%
From the Hugging Face model README
Stage-1 endpoint — hidden-state alignment. Exit: rel-MSE 0.0077, wikitext PPL 52.9 (1.9x teacher). Accepted via KD-smoke functional gate (val KL 1.554->0.563 in 2 steps); raw-prose degeneracy at this stage is MiMo-specific calibration, fixed by KD.
Part of the MiMo second-model arm of a study converting full-attention models into GDN (gated DeltaNet) hybrids (uniform 1:4 retention) and recovering capability via staged distillation. Teacher/base: MiMo-7B-RL-0530.
NOTE: these checkpoints use the custom mimo_gdn architecture — register it
before loading (import src.models.mimo_gdn from the project repo, or use
the load_causal_lm helper in src/eval/sample_generations.py). Bare
AutoModelForCausalLM will not work. vLLM serving needs
src.models.vllm_mimo_gdn.register().
Pipeline: surgery/init (uniform 1:4 plain inheritance) -> stage-1 hidden-state alignment -> stage-2a forward-KL KD @4k -> stage-2b long-context KD @32k. Within-arm reference: teacher wikitext-2 PPL 27.51 (RL'd teacher; never cross-arm-compare absolute PPL).