mini-glm-5.2

Mini GLM 5.2 Active-MTP Debug Checkpoint

This is a small GLM-5.2-style checkpoint built for inference plumbing and kernel debugging, not for model quality.

The base model is a 14-layer mini GLM-MoE-DSA model trained for readable English. It has been upscaled to GLM-5.2-compatible tensor dimensions and includes one appended MTP/EAGLE next-token layer for SGLang speculative decoding.

Intended Use

• Debug SGLang GLM-MoE-DSA inference without loading the full GLM 5.2 model.
• Exercise DSA, MoE, attention, TP sharding, FP8 KV cache, and EAGLE/MTP paths.
• Validate kernel changes against a checkpoint that is small in layer count but keeps GLM-5.2-like shapes.

This model is expected to produce readable but repetitive English. It is not a general-purpose language model.

MTP Details

The appended MTP layer is active and nonzero. Its attention and MoE tensors were trained, rather than zeroed out, so this checkpoint can exercise the MTP decoder attention/MoE path during serving.

The MTP trainer uses SGLang EAGLE nextn alignment:

hidden[t] + token[t+1] -> token[t+2]

For rollout step k:

input token = token[t+k+1]
target      = token[t+k+2]

This is intentionally not the standard LM shift hidden[t] + token[t] -> token[t+1].

Compatibility

SGLang

This directory is intended for SGLang serving with GLM-MoE-DSA, TileLang DSA, FP8 KV cache, and EAGLE speculative decoding.

Example shape of the serving command:

source ./ENV_RUN.sh
CUDA_VISIBLE_DEVICES=6,7 HIP_VISIBLE_DEVICES=6,7 sglang serve \
  --model-path /root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang \
  --tp 2 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 4096 \
  --max-total-tokens 4096 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --disable-shared-experts-fusion \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled

Hugging Face Transformers

The base model is loadable with Hugging Face Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)

HF inference uses only the base 14-layer causal LM. The appended MTP tensors are reported as unexpected keys because HF Transformers' glm_moe_dsa model does not implement the MTP/EAGLE nextn path.

Training Summary

The MTP layer was trained from copied active decoder weights with the corrected SGLang EAGLE alignment on a small SGLang-served token dataset.

Final offline served-data metrics for the source MTP checkpoint:

These are offline metrics on the small served-token dataset and should be treated as a debugging signal, not a model-quality benchmark.

Example Outputs

The model is only expected to produce barely passable English. Repetition, generic phrasing, weak reasoning, and topic drift are normal.

These examples show the intended quality bar: readable text for plumbing tests, not useful answers.

Example 1

Prompt:

The quick brown fox

Observed HF base-model response:

The quick brown fox is a very good thing to do. It is a good thing to do.

Example 2

Prompt:

Write one short paragraph about a library in a small town.

Expected style of response:

The library is a quiet place in the center of the town. People come there to
read books, ask questions, and sit at the tables in the afternoon. It is not a
large building, but it is useful for the people who live nearby.

Example 3

Prompt:

Explain why rain falls from clouds.

Expected style of response:

Rain falls from clouds when water in the air becomes heavy enough to fall down.
The cloud is made of small drops, and those drops can join together. When they
become too heavy, they fall as rain.

Example 4

Prompt:

Continue this sentence: The old machine started slowly because

Expected style of response:

The old machine started slowly because the parts were worn and the motor needed
time to move. It made a small sound, then a louder sound, and finally began to
work again.

The responses above are coherent enough to inspect inference behavior, token flow, speculative decoding, and kernel changes. They should not be used to judge instruction following or factual accuracy.