GLM-5.2-2bit-MoE-planes-pruned208-tp2

GLM-5.2 · 2-bit MoE expert planes · pruned 256→208 · TP2 for 2× DGX Spark

GLM-5.2 (753B) running at 15 tok/s sustained decode on two NVIDIA DGX Sparks (GB10, 128 GB unified LPDDR5X each, linked over 200G RoCE). This repo contains the custom-format expert weights that make that possible:

• 2-bit packed MoE expert planes (w13 / w2 per layer), prepacked from the FP8 checkpoint, sharded for tensor-parallel 2 (rank0/ + rank1/).
• Expert-pruned 256→208 per MoE layer: the 48 coldest experts per layer (by routing traffic measured over 5.3k tokens across 12 domains at native top-k=8) are dropped, REAP-style — pruned experts are masked out of selection only; gate weights renormalize over the kept experts. This shrinks the planes from 97 → 79 GB per rank, which is the whole trick: 79 GB fits each node's page cache, so decode runs fault-free from RAM instead of thrashing NVMe.
• The MTP drafter layer (layer 78) is kept unpruned (all 256 experts), enabling speculative decoding (num_speculative_tokens=1, ~68% acceptance) for the final 11.4 → 15.0 tok/s step.

Routing stays at the model's native top-k=8 — this is not a low-k quality trade. Quality checks (arithmetic, factual, prose-coherence battery + probes): clean. Measured eval scores vs the unpruned base are being run and will be added here.

What you need

Layout

rank0/   layer_XXX.{planes13,planes2,sc13,sc2}.npy + layer_XXX.meta.json   (TP rank 0)
rank1/   same files for TP rank 1
keep208.json   kept-expert ids per layer (208 of 256; layer 78 keeps all 256)

Each meta.json carries geometry, tp_rank, and the keep list the loader uses to mask router bias and remap expert ids to compacted rows (cudagraph-safe LUT gather).

Serving (2× DGX Spark)

# on each node: base checkpoint at ~/models/hf/GLM-5.2-FP8,
# this repo's rankN/ at ~/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208
bash spark/start-ray-cluster.sh   # head node
VLLM_MOE_W2_PREPACKED_DIR=$HOME/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208 \
  MTP_K=1 bash spark/serve-glm52-tp2-mtp.sh

Measured on this setup (512→1024-token differential, post-settle, essay domain): 15.0 tok/s sustained, fault-free residency (≤ tens of MiB NVMe reads per kilotoken).

How the speed was found (short version)

1. 2-bit expert planes + TP2 across both Sparks: 4.0 tok/s at k=8 — bound by page-cache thrash (97 GB planes > ~88 GB cache), not LPDDR bandwidth.
2. Measured real routing traffic: GLM-5.2 routing is FLAT (50% of traffic needs 27% of (layer, expert) cells), but pruning the coldest 48/layer costs only ~6% of routed slot traffic — and buys full residency: 11.4 tok/s, quality clean.
3. MTP speculative decode — a 17% loss on thrashing planes — flips to +31% once verify reads come from page cache: 15.0 tok/s.

Full session-by-session engineering log: spark/handoffs/ in the GitHub repo.

Caveats

• Expert selection is frequency-based (traffic-cold), not saliency-based (REAP-scored); a saliency-picked set may be slightly better. Exposure: ~5.9% of routed slot traffic renormalizes onto kept experts (worst layer 12%).
• The 2-bit planes require the vLLM-Moet fork's moe_w2 kernel path (SM 12.1 / GB10 cubins included in the repo). This is not a GGUF / standard-vLLM artifact.
• TP2-specific sharding; other world sizes need re-prepacking from the base checkpoint (spark/prepack_planes.py + spark/routing/prune_planes.py).