Motif-3-Beta-FP8

Motif-3-Beta-FP8

Warning: the upstream HF inference path is broken (acknowledged by the vendor on 2026-07-22: "HF.generate currently has some issues"). The original checkpoint produces incoherent output before any quantization is applied; the released modeling code and weights most likely belong to different revisions. This repository is a reference implementation of expert-only block-FP8 quantization for research purposes, not a usable text-generation model until upstream ships matching code.

Expert-only FP8 (E4M3, 128x128 block) quantization of Motif-Technologies/Motif-3-Beta (commit 32b0305): 630 GB BF16 -> 322 GB, with 97.84% of parameters quantized and routing decisions bit-identical to the original.

Model Overview

Quantized: moe.experts.gate_up_proj and moe.experts.down_proj of all 51 MoE layers (308.0B parameters). Preserved in BF16: attention (GDLA), mHC, per-expert PolyNorm coefficients, router, shared experts, dense MLP, MTP, embeddings, lm_head.

Fidelity

All comparisons are FP8 (dequantized) vs. the original BF16 checkpoint.

The error is uniform across layers -- no fragile layers or experts.

End-to-end: cannot be certified -- the BF16 baseline itself is broken upstream. Under an identical serving pipeline (2x H100), original BF16 scores NLL 20.06 and this FP8 scores 20.95, failing identically. The breakage is therefore attributable to the upstream checkpoint, not to quantization.

Note: e2e figures were measured under Transformers' default grouped_mm experts backend, which is affected by the dispatch bug below (expert 0's PolyNorm coefficients applied to all experts) -- treat them as indicative only. The weight / expert / layer results above are unaffected: the layer test instantiated the MoE module standalone, which falls back to the eager path (correct per-expert coefficients, confirmed via the dispatch-fallback warning in our logs).

Known Upstream Issues

Found during verification (transformers 5.14.1, snapshot 32b0305):

• Rotary embedding bug crashes the stock HF forward path on any input (yarn init ignores rope_head_dim; 192-dim cos/sin vs. 64-dim q_pe).
• sliding_window silently ignored by the sdpa/eager attention paths.
• Transformers' default grouped_mm/batched_mm MoE backends call _apply_gate() without an expert index, so every expert silently uses expert 0's per-expert PolyNorm coefficients (found in base repo discussion #6; workaround: experts_implementation="eager").
• Config keys such as polynorm_output_scale and hidden_clamp are consumed by the vendor's native vLLM implementation but absent from the released HF modeling code -- direct evidence of the revision mismatch.
• Suspected embedding-scale mismatch between checkpoint and code.
• Non-standard MoE config key names break automatic recognition by serving frameworks.

The bundled modeling_motif.py patches the crash and adds vLLM compatibility; the config includes alias keys and a corrected auto_map.

Usage

Stock from_pretrained with the FP8 config fails: the Transformers FP8 loader replaces expert modules with a generic implementation that does not support this model's per-expert Grouped PolyNorm activation. Serving requires custom expert handling (dequantize routed experts on demand) -- see tools/stream_forward.py.

The vendor's official vLLM image (ghcr.io/motiftechnologies/vllm:v0.20.2-motif3.rc2) uses the same recipe (expert-only 128x128 E4M3, --quantization modelopt_blockfp8) but quantizes BF16 weights online at load time and does not accept pre-quantized checkpoints like this one.

# Weight-level dequantization
import torch
from safetensors import safe_open

with safe_open("model-00105-of-00155.safetensors", framework="pt") as f:
    w = f.get_tensor("model.layers.2.moe.experts.gate_up_proj")           # fp8 e4m3
    s = f.get_tensor("model.layers.2.moe.experts.gate_up_proj_scale_inv") # fp32

E, O, I = w.shape
deq = (w.float().reshape(E, O // 128, 128, I // 128, 128)
        * s[:, :, None, :, None]).reshape(E, O, I).to(torch.bfloat16)

Verified on 2x NVIDIA H100 80GB + 442 GB RAM: the full checkpoint fits in VRAM + system RAM with tiered expert residency (activated experts fetched and dequantized per token).

Repository Contents

License

Inherits the original model license: personal, educational, and non-commercial research use only. Commercial use requires prior written permission from Motif Technologies.