GLM-4.6-Flash-text

GLM-4.6-Flash-text

zai-org/GLM-4.6V-Flash with the vision stack removed. No fine-tuning, no distillation, no retraining. The 181 vision tensors were deleted and the remaining 523 re-keyed onto Glm4ForCausalLM.

The text path is bit-identical to the original. See Verification.

Removed: 892,498,432 params, 8.671% of the model, 1.785 GB.

What you get

The vision tower is 8.671% of this checkpoint. Removing it saves 1.79 GB out of 20.59. If you want a smaller model, this isn't much of one.

The useful part is what that turns into on a card. Both models served on one A100-SXM4-40GB under vLLM 0.27.1 at --max-model-len 32768:

The weights account for less than a third of that cache gain. vLLM's own memory profile splits it up:

The 4.82 GiB is vLLM profiling the vision path. It measures one maximum-size video, sizes a 66,242-token encoder cache, and holds the result for the life of the server whether or not an image is ever sent. Both checkpoints store KV at the same 40,960 bytes per token, so all of the extra cache here is freed VRAM rather than a cheaper cache.

Much of that is reachable without this model. Serving the original with --limit-mm-per-prompt '{"video": 0}' drops its peak activation to 0.97 GiB and its encoder cache to 6,084 tokens:

So the honest figure depends on the comparison. Against the original as it ships, +54%. Against an original already configured for text-only serving, +16%. What no flag reaches is the 1.73 GiB of weights, and a config with no multimodal settings to get wrong.

Turning both modalities off is not an option. --limit-mm-per-prompt '{"video": 0, "image": 0}' makes vLLM 0.27.1 skip loading the tower and then fail in profiling with AttributeError: 'NoneType' object has no attribute 'size'.

Architecture

GLM-4.6V-Flash is a bolted-on vision tower: a 24-layer ViT feeding soft tokens into the decoder's embedding stream via masked_scatter. Text tokens never touch a vision weight, so deleting the branch removes one edge from the graph and leaves the text computation alone.

What remains is a standard GLM-4 decoder: 40 layers, 4096 hidden, 13696 intermediate, 32 attention heads with 2 KV heads (GQA 16:1), 151552 vocab, 131072 max positions, untied lm_head.

Verification

The real risk here was RoPE. GLM-4.6V-Flash uses multimodal RoPE (mrope_section: [8, 12, 12], summing to 32 = head_dim 128 × partial_rotary_factor 0.5 ÷ 2), which splits the rotary budget across temporal, height and width axes. For text-only input all three axes carry the same position index, so mRoPE should collapse to standard RoPE. That's an argument though, not a measurement.

Logits compared against the original on identical text input:

max|d|=0.000e+00  argmax_match=True  'The capital of France is'
max|d|=0.000e+00  argmax_match=True  'def fibonacci(n):'
max|d|=0.000e+00  argmax_match=True  'Explain why the sky appears blue, in one sentence.'
max|d|=0.000e+00  argmax_match=True  '1, 1, 2, 3, 5, 8, 13,'
max|d|=0.000e+00  argmax_match=True  'Translate to German: The weather is cold today.'
max|d|=0.000e+00  argmax_match=True  'The three laws of thermodynamics state that'

EXACT MATCH -- mRoPE collapsed to RoPE cleanly. Extraction is lossless.

Short prompts don't exercise RoPE at depth, which is where a position-encoding bug would show up, so the same check was run at length:

sequence length: 1207
long-context max|d| = 0.000e+00
PASS

Zero divergence at 1,207 tokens. On text this is the original model.

Files

bf16 safetensors at the root, quantizations under gguf/.

All five load and generate coherently under llama.cpp (architecture: glm4, 131072 context). Sizes are GB (10⁹ bytes) as the Hub reports them; ls -h will show smaller GiB numbers for the same files.

If you only want GGUF, the same quantizations sit at the repo root of sartajbhuvaji/GLM-4.6-Flash-text-GGUF, where the Hub's quantization picker renders and llama-cli -hf / ollama run hf.co/… resolve directly. The gguf/ copies here are identical, so use whichever is convenient.

There is no NVFP4 build. NVFP4 is NVIDIA's Blackwell format (E2M1, 16-element blocks, FP8 E4M3 block scales) and needs SM100+ hardware to quantize and serve. It is not a GGUF quant and can't be produced on Ampere. If you want one, run LLM Compressor on a B200 with the bf16 weights here as input.

Usage

Every command below was run end to end on a single A100-SXM4-40GB at the versions pinned in each block. Measured numbers are in Benchmarks.

vLLM

pip install vllm==0.27.1 ninja

vllm serve sartajbhuvaji/GLM-4.6-Flash-text \
  --served-model-name GLM-4.6-Flash-text \
  --max-model-len 32768 \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 --enable-auto-tool-choice

That works as-is. It pulls only the 18 GB of safetensors and ignores the gguf/ folder in this repo. Serving takes ~120 s from nothing on a fast link including download, ~45 s with weights cached and the compile cache warm. At --max-model-len 32768 on a 40 GB card there's still a 479,280-token KV cache left, so raise the context freely.

If you hit FileNotFoundError: ninja: vLLM's compile path shells out to ninja, and the failure surfaces three frames deep as RuntimeError: Engine core initialization failed, with the real cause buried far above it in the log. pip install ninja fixes it, but only if ninja is on your PATH. Invoking /path/to/venv/bin/vllm directly does not put that venv's bin on PATH; only activating the venv does.

SGLang

pip install "sglang[all]==0.5.18"

python -m sglang.launch_server \
  --model-path sartajbhuvaji/GLM-4.6-Flash-text \
  --served-model-name GLM-4.6-Flash-text \
  --context-length 32768 \
  --reasoning-parser glm45 --tool-call-parser glm45 \
  --host 0.0.0.0 --port 30000

--reasoning-parser auto also works, reading the parser choice off the chat template.

Calling either server

Both expose the OpenAI API, so the same client works against either. Only the port differs (vLLM 8000, SGLang 30000).

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-4.6-Flash-text",
    "messages": [{"role": "user", "content": "What is a mixture-of-experts layer?"}],
    "max_tokens": 900
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="GLM-4.6-Flash-text",
    messages=[{"role": "user", "content": "What is a mixture-of-experts layer?"}],
    max_tokens=900,
)
print(r.choices[0].message.content)

Tool calling works on both with the glm45 tool parser above. Pass tools=[...] and read message.tool_calls as usual.

transformers

pip install torch==2.13.0 transformers==5.15.1 accelerate

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sartajbhuvaji/GLM-4.6-Flash-text", dtype=torch.bfloat16, device_map="auto"
)
tok = AutoTokenizer.from_pretrained("sartajbhuvaji/GLM-4.6-Flash-text")

msgs = [{"role": "user", "content": "What is a mixture-of-experts layer?"}]
enc = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**enc, max_new_tokens=900)
print(tok.decode(out[0][enc["input_ids"].shape[1] :], skip_special_tokens=True))

Needs a transformers version carrying Glm4ForCausalLM (v4.52+). Keep max_new_tokens around 900, see Output format.

llama.cpp

Use the GGUF repo, where the files sit at the root and -hf resolves them:

llama-cli -hf sartajbhuvaji/GLM-4.6-Flash-text-GGUF:Q4_K_M \
  -p "Explain gradient descent" -n 900 -ngl 99 -st

-st (--single-turn) matters for scripted use. Without it llama-cli drops into interactive mode and waits on stdin, which looks like a hung GPU. The older -no-cnv flag has been removed. -ngl 99 offloads every layer to the GPU.

To use the copies in this repo instead, download by explicit path, since -hf only resolves root-level GGUFs:

hf download sartajbhuvaji/GLM-4.6-Flash-text \
  gguf/GLM-4.6-Flash-text-Q4_K_M.gguf --local-dir .
llama-cli -m gguf/GLM-4.6-Flash-text-Q4_K_M.gguf -p "Explain gradient descent" -n 900 -ngl 99 -st

There is no prebuilt Linux CUDA binary; llama.cpp publishes CUDA archives for Windows only. On Linux, build it:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_CURL=ON
cmake --build build -j --target llama-cli llama-server

(80 is A100, 89 for L40S/4090, 90 for H100.)

Ollama

ollama run hf.co/sartajbhuvaji/GLM-4.6-Flash-text-GGUF:Q4_K_M

Output format

This is a reasoning model. Its output has a few quirks.

It thinks first, at length, and often in Chinese. Every answer is preceded by a <think>…</think> block, frequently in Chinese no matter what language you prompted in. Budget for it: max_tokens=160 reliably returns a truncated monologue with no answer in it, which looks like a broken model. 900 is a safe default.

The final answer is wrapped in <|begin_of_box|>…<|end_of_box|>, a GLM convention that survives into this checkpoint. No parser strips it, so strip it yourself:

import re

answer = re.sub(r"<\|(begin|end)_of_box\|>", "", content).strip()

The reasoning field has two different names. With a reasoning parser enabled the thinking is split out of content into its own field, but vLLM 0.27.1 calls it message.reasoning while SGLang calls it message.reasoning_content. Client code that hardcodes one silently drops the reasoning on the other. Read both:

m = r.choices[0].message
reasoning = getattr(m, "reasoning", None) or getattr(m, "reasoning_content", None)

Turning thinking off works on both servers via the chat template, and makes a large difference: the France question drops from 53 tokens to 2.

{"chat_template_kwargs": {"enable_thinking": false}}

Benchmarks

512-token prompts, 256-token outputs, greedy, with ignore_eos pinning every request to exactly 256 output tokens so the runs are comparable. Both engines were driven through the same OpenAI-compatible streaming client, so a gap between them is a gap between the engines and not between two benchmark tools. Every row is warmed: one full pass discarded before measuring. Raw JSON and the environment capture are in the unfuse repo.

1× NVIDIA A100-SXM4-40GB, sm_80, driver 580.105.08, CUDA 13.0, 400 W, torch 2.13.0+cu130, Ubuntu 24.04.4, kernel 6.8.0-1046-nvidia.

Notes on the numbers

This model is not faster than the original. Every concurrency-1 row lands between 61.8 and 63.7 tok/s and every concurrency-16 row between 962.7 and 994.9, a 3% spread across eight measurements of two checkpoints on two engines. That is the expected result: the same 40 text layers do the work in every row, and both checkpoints sit against the same ceiling, see Bandwidth limit. A run showing this model ahead by more than a few percent would be measuring the harness.

The case for the split is the cache headroom in What you get and a serving path with no multimodal configuration in it. It is not throughput.

Scheduler limits, because two rows are only comparable if the engine offered both models the same number of slots. vLLM gave both max_num_seqs=256, clamped down from max_num_batched_tokens=2048; it prints this only under VLLM_LOGGING_LEVEL=DEBUG, so it was read from a separate start with identical flags rather than assumed. SGLang derives its own and gave the original 3,085 and this model 4,096. Those differ, but both sit far above the 16 concurrent requests tested, so neither run was throttled. SGLang also picked a lower static memory fraction for the multimodal model, 0.692 against 0.823, which is why its KV pools differ more than vLLM's: 197,483 tokens against 380,129.

Warmup, because one row moved by 24.9%. SGLang on this model read 50.7 tok/s at concurrency 1 on the first pass and 63.3 on the second, with TPOT identical at 15.58 ms both times, so the whole gap was the first request paying startup. Set against SGLang's warmed original at 63.7, the cold number would have reported this model as 20% slower than its parent. Discard a pass, and never compare a warmed row against an unwarmed one. The other seven rows moved by less than 0.3%.

Each engine also ships its own benchmark tool, and run against those on an earlier pass of this model, SGLang appeared to win TTFT by 3×. They disagree on prompt sampling, warmup and what counts as duration, so their numbers cannot go in one table. Every row above comes from one client hitting both servers over the same HTTP path, which reversed that result.

Bandwidth limit

At concurrency 1 a dense model is memory-bandwidth bound. Every weight is read from HBM once per token, so the bus sets a ceiling no engine flag can beat.

Both models read the same bytes. Decode touches only the text path, and the original's vision tower sits in VRAM without being streamed when the prompt carries no image, so the tower costs residency rather than bandwidth. Charging the original for all 20.59 GB would lower its ceiling by 9% and invent an advantage for this model that the measurements do not show.

An A100-SXM4-40GB has 1.555 TB/s of HBM2e. Reading 18.80 GB of weights costs at least 12.09 ms per decoded token, a floor of 82.7 tok/s, and the best measured 63.3 sits at 77% of it.

Single-stream decode here is already close to the hardware limit. There is roughly 20-25% of headroom and no server flag will find more than that; faster single-stream generation needs more bandwidth or fewer bytes, which is what the Q4_K_M build below does. It also explains the flat concurrency-1 column above: every configuration is moving a near-identical number of bytes over the same bus, and the engines only diverge once concurrency amortises those reads.

Treat this as ±10% rather than a precise efficiency figure. It counts weight traffic only and ignores KV-cache reads, which grow with context and are not free.

Other stacks

transformers and llama.cpp are measured on this model only. The original has no GGUF build to compare against, and transformers does no continuous batching, so neither produces a row that belongs in the table above.

transformers is about 3× slower per stream, which is fine for scripting and wrong for serving.

llama.cpp is a different quantization (llama-bench, Q4_K_M, all layers offloaded):

Q4_K_M decodes about 2× faster than bf16 on the same card at a third of the memory, the usual quantization trade, and the reason it's the recommended file for single-stream use.

Startup, cold, from a warm page cache: vLLM served this model in 55 s against the original's 185 s, and SGLang in 60 s against 120 s. Most of that difference is vLLM's multimodal profiling pass, 116 s of engine init against 11 s. transformers loads the weights in 5 s.

Reproducing this

from safetensors.torch import load_file, save_file

# 1. drop every tensor under model.visual.  (181 tensors, 892,498,432 params)
# 2. rename  model.language_model.*  ->  model.*
# 3. keep lm_head.weight -- it is UNTIED (tie_word_embeddings: false)
sd = {
    k.replace("model.language_model.", "model.", 1): v
    for k, v in load_file(shard).items()
    if not k.startswith("model.visual.")
}

Then rebuild the config through Glm4Config, dropping mrope_section and vision_config, and set architectures = ["Glm4ForCausalLM"]. Glm4Config does not populate that field, and without it AutoModelForCausalLM has nothing to dispatch to.

Vocabulary needs no work: vocab_size stays 151552, and the image/video token ids (151363/151364) remain in it. They're simply never emitted.

Limitations

• No vision. Passing images does nothing, since the tokens have no embedder behind them. Use the original model if you need multimodal.
• Reasoning model. It emits <think> blocks before answering, sometimes in Chinese regardless of prompt language. Budget max_new_tokens accordingly; 160 is not enough to get past the reasoning to an answer.
• Inherits everything else from GLM-4.6V-Flash, including its biases and knowledge cutoff. Text behaviour is bit-identical, so any evaluation of the original's text ability transfers exactly.
• The headline cache figure depends on what you compare against. Against the original as it ships, +54% KV tokens; against an original served with --limit-mm-per-prompt '{"video": 0}', +16%. See What you get.
• Benchmarks cover one GPU and one shape: 512-in/256-out on a single A100-SXM4-40GB at concurrency 1 and 16. Other context lengths, batch shapes or hardware will rank the engines differently.
• Quantization is not verified bit-exact. The bit-exactness result above covers the bf16 weights only. The GGUF quants are lossy by construction; each was checked to load and generate coherent text under llama.cpp, but no perplexity or benchmark comparison against bf16 was run. If you need a measured quality delta, compute it yourself.

Provenance

Derived from zai-org/GLM-4.6V-Flash (MIT). This model is MIT as well.

Conversion and verification were run on a single A100-SXM4-40GB with transformers 5.16.0.dev0 and torch 2.7.0. GGUF builds used llama.cpp at master with the GLM4 architecture.

Benchmarks were run separately on a single A100-SXM4-40GB with vLLM 0.27.1 and SGLang 0.5.18, both on torch 2.13.0+cu130. The harness, the raw result JSON and the engine startup logs behind every memory figure are in github.com/SartajBhuvaji/unfuse under notebooks/glm-4.6v-flash/bench/.