llama-3.1-8b-instruct-4x1-moe-switch-top1

Llama 3.1 8B Instruct 4x1 MoE — Switch-style top-1 router

This checkpoint reuses the exact weights of Jongbin-kr/llama-3.1-8b-instruct-4x1-moe but changes the versioned custom model implementation and routing configuration:

• four cloned Llama FFN experts per Transformer layer;
• one active expert per token;
• the selected top-1 gate retains its original softmax probability instead of being renormalized to one;
• Switch-style load-balancing coefficient 0.01;
• ST-MoE router z-loss coefficient 0.001 during training.

Because the gate implementation changes the forward pass, this is a distinct base model rather than a new revision of the original normalized-gate model. Load it with trust_remote_code=True.