mini-glm-5.2-fp8

Mini GLM 5.2 Active-MTP Debug Checkpoint, FP8

This is a small GLM-5.2-style checkpoint built for inference plumbing and kernel debugging, not for model quality. This directory is the FP8 weight-quantized variant of the active-MTP full-dimension checkpoint.

The base model is a 14-layer mini GLM-MoE-DSA model trained for readable English. It has been upscaled to GLM-5.2-compatible tensor dimensions and includes one appended MTP/EAGLE next-token layer for SGLang speculative decoding.

The FP8 export follows the public GLM-5.2-FP8 checkpoint convention:

• eligible 2D linear weights are stored as torch.float8_e4m3fn
• each quantized weight has a sibling *_scale_inv tensor
• scale tensors are FP32 with 128x128 block granularity
• embeddings, norms, router gates, DSA weights_proj, and MTP glue projections/norms remain unquantized

Intended Use

• Debug SGLang GLM-MoE-DSA inference without loading the full GLM 5.2 model.
• Exercise DSA, MoE, attention, TP sharding, FP8 KV cache, and EAGLE/MTP paths.
• Validate kernel changes against a checkpoint that is small in layer count but keeps GLM-5.2-like shapes.

This model is expected to produce readable but repetitive English. It is not a general-purpose language model.

MTP Details

The appended MTP layer is active and nonzero. Its attention and MoE tensors were trained, rather than zeroed out, so this checkpoint can exercise the MTP decoder attention/MoE path during serving.

The MTP trainer uses SGLang EAGLE nextn alignment:

hidden[t] + token[t+1] -> token[t+2]

For rollout step k:

input token = token[t+k+1]
target      = token[t+k+2]

This is intentionally not the standard LM shift hidden[t] + token[t] -> token[t+1].

Compatibility

SGLang

This directory is intended for SGLang serving with GLM-MoE-DSA, TileLang DSA, FP8 KV cache, and EAGLE speculative decoding.

Example shape of the serving command:

source ./ENV_RUN.sh
CUDA_VISIBLE_DEVICES=6,7 HIP_VISIBLE_DEVICES=6,7 sglang serve \
  --model-path /root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang-fp8 \
  --tp 2 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 4096 \
  --max-total-tokens 4096 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --disable-shared-experts-fusion \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled

Hugging Face Transformers

Use the BF16/full-dimension checkpoint for Hugging Face Transformers CPU/GPU reference inference. This FP8 directory is intended for SGLang's GLM FP8 loader. The FP8 tensors match the GLM-5.2-FP8 safetensors layout, but generic HF Transformers inference may not dequantize this custom GLM-MoE-DSA FP8 format.

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)

HF inference uses only the base 14-layer causal LM. The appended MTP tensors are reported as unexpected keys because HF Transformers' glm_moe_dsa model does not implement the MTP/EAGLE nextn path.

Training Summary

The MTP layer was trained from copied active decoder weights with the corrected SGLang EAGLE alignment on a small SGLang-served token dataset.

Final offline served-data metrics for the source MTP checkpoint:

These are offline metrics on the small served-token dataset and should be treated as a debugging signal, not a model-quality benchmark.

Example Outputs

The model is only expected to produce barely passable English. Repetition, generic phrasing, weak reasoning, and topic drift are normal.

Observed SGLang FP8 raw-completion sample:

Prompt:

A small language model is useful for testing inference code because

Response:

it is an independent scientific method. However, the application of the
language is considered as a social, practical approach. It has a wide range of
specific characteristics and may be used for clinical practice.

The text is readable but semantically loose; that is the expected quality bar for this plumbing checkpoint.

These examples show the intended quality bar: readable text for plumbing tests, not useful answers.

Example 1

Prompt:

The quick brown fox

Observed HF base-model response:

The quick brown fox is a very good thing to do. It is a good thing to do.

Example 2

Prompt:

Write one short paragraph about a library in a small town.

Expected style of response:

The library is a quiet place in the center of the town. People come there to
read books, ask questions, and sit at the tables in the afternoon. It is not a
large building, but it is useful for the people who live nearby.

Example 3

Prompt:

Explain why rain falls from clouds.

Expected style of response:

Rain falls from clouds when water in the air becomes heavy enough to fall down.
The cloud is made of small drops, and those drops can join together. When they
become too heavy, they fall as rain.

Example 4

Prompt:

Continue this sentence: The old machine started slowly because

Expected style of response:

The old machine started slowly because the parts were worn and the motor needed
time to move. It made a small sound, then a louder sound, and finally began to
work again.

The responses above are coherent enough to inspect inference behavior, token flow, speculative decoding, and kernel changes. They should not be used to judge instruction following or factual accuracy.