GLM-5.3-Flash-Uncensored-NVFP4-Pruning-62GB

<div align="center">

GLM-5.3-Flash · Uncensored · NVFP4 · Agentic Expert-Pruned

92 of 288 routed experts per layer · ~107B total / ~18B active parameters · 62 GiB · fits one 96 GB GPU

Pruned for agents, tool calling and code — not for trivia.

</div>

Revision 2 (2026-10-13). This repository was rebuilt from scratch. The previous revision (80 experts, ~62 GiB) was selected with weight-norm heuristics on ~200 English tokens and lost multilingual and tool-calling ability; it is still available in the git history. The repo name keeps "62GB" for link stability — the weights are now 62 GiB.

What is this

An expert-pruned version of orcarouter/GLM-5.3-Flash-Uncensored-NVFP4 (abliterated GLM-5.3-Flash, 314B MoE, NVFP4 experts) that fits entirely in the memory of a single 96 GB GPU (tested on an RTX PRO 6000 Blackwell 96 GB; the bundled engine uses 67.5 GiB after loading and ~76 GB while generating, so 80 GB cards are very tight and untested).

To fit more experts into ~62 GiB, the large attention projections (KDA q/k/v/o, MLA q_b/o), the shared experts and the dense MLPs of layers 0–2 are stored as FP8 E4M3 with per-output-channel scales (compressed-tensors float-quantized, near-lossless) instead of BF16. This frees ~6.6 GiB = 12 extra experts per layer compared with a plain BF16 build of the same size.

Untouched: routed experts' NVFP4 weights (packed bytes copied bit-exact; router rows sliced and re-indexed), small attention projections, norms, hyper-connections, vision tower, tokenizer and chat template. The MTP speculative head is removed.

How the experts were chosen

1. Exact layer-wise forward of the full 191 GiB model on one GPU (one decoder layer resident at a time, NVFP4 experts dequantized on the fly) over a domain-tagged calibration set: SWE-agent trajectories, Hermes/Glaive function calling, the same function-calling dialogues rendered in GLM's native tool-call format (tools=, <tool_call>…<arg_key>…, tool results), Magicoder/CodeFeedback code, OpenCodeReasoning traces, RU chat (GrandMaster-PRO-MAX), EN chat (UltraChat) and some math — 284 calibration sequences × 2048 tokens.
2. REAP saliency per expert and domain: Σ over routed tokens of gate_weight · ‖expert_output‖₂, normalized per layer.
3. Priority-weighted selection per layer — native tool calling 0.20, SWE trajectories 0.20, code 0.18, code reasoning 0.10, RU chat 0.10, EN chat 0.10, Hermes tool text 0.06, math 0.06 — top-92 kept. Knowledge-heavy domains (science, encyclopedic facts, rare languages) were deliberately not protected.

The kept expert indices for every layer are in keep.json.

Evaluation

Held-out perplexity (60 sequences × 2048 tokens never used for selection)

Lower perplexity than the full model on some domains does not mean the model got smarter — top-1 agreement shows it does deviate from the full model. Treat these as "nothing broke" signals, not as benchmark scores.

Functional agent tests (real tools, objective success checks, temperature 0)

Tools: list_dir, read_file, write_file, run_python, run_tests; the agent loop feeds real tool results back.

Small, sanity-level tests — run your own benchmarks (SWE-bench, BFCL, LiveCodeBench, …) before relying on it.

Usage

Tested: bundled single-GPU engine (OpenAI-compatible API)

inference/ contains the engine used for all tests above: plain PyTorch + 🤗 Transformers modules, NVFP4 experts kept packed on the GPU and dequantized per token. Requires transformers>=5.16, torch>=2.11, safetensors; flash-linear-attention strongly recommended.

huggingface-cli download Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruning-62GB --local-dir glm-agentic
mkdir -p /tmp/work && cp glm-agentic/inference/*.py /tmp/work/
python /tmp/work/serve.py glm-agentic     # loads in ~1-2 min, serves http://127.0.0.1:8000/v1/chat/completions

import json, urllib.request
req = {"messages": [{"role": "user", "content": "Какая погода в Казани?"}],
       "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object",
                 "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
       "max_tokens": 300}
r = urllib.request.urlopen(urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
        data=json.dumps(req).encode(), headers={"Content-Type": "application/json"}))
print(json.load(r)["choices"][0]["message"])   # -> tool_calls: get_weather(city="Казань")

Speed with this engine: ~4 tokens/s decode on an RTX PRO 6000 (it is a reference implementation, not a fast kernel).

vLLM (tested, ~46.7 tokens/s single stream, ~176.5 tokens/s with 8 parallel requests)

The checkpoint is stored in the layout of the bundled engine, so vLLM (0.31, which natively supports Glm5NextForConditionalGeneration) needs a one-time conversion (~5 min, writes ~69 GiB). inference/convert_vllm.py does it and fixes four things that otherwise produce a crash or NaN logits (every one of them was observed while bringing this model up):

1. FP8 attention / shared-expert / dense-MLP weights → BF16. vLLM's Glm5Next loader expects block-wise FP8 attention scales and expands per-channel scales to a 1 TiB tensor; its FP8 linear kernel also needs an NVRTC JIT build.
2. Per expert, gate_proj and up_proj have different NVFP4 global scales (up to 2×); vLLM fuses them into one w13 tensor and needs a single scale. The converter sets both to min(Gg, Gu) and rescales the block scales of the other projection.
3. input_activations is removed from the quantization config. The checkpoint is weight-only (it has no input scales); with activation quantization declared, vLLM takes the W4A4 path, turns the missing scales into inf and returns NaN → garbage text. Without it vLLM uses the W4A16 (Marlin) path.
4. The sandbox needs the CUDA headers of the same version as nvcc for DeepGEMM's JIT (nvidia-cuda-cccl==<nvcc version>).

pip install uv && uv venv --python-preference only-managed -p 3.12 /tmp/vv
uv pip install --python /tmp/vv/bin/python vllm instanttensor
V=$(/tmp/vv/bin/python -c "import flashinfer;print(flashinfer.__version__)")
uv pip install --python /tmp/vv/bin/python "flashinfer-jit-cache==$V" --extra-index-url https://flashinfer.ai/whl/cu130

huggingface-cli download Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruning-62GB --local-dir glm-src
python glm-src/inference/convert_vllm.py glm-src glm-vllm          # any python with torch + safetensors

/tmp/vv/bin/vllm serve glm-vllm --served-model-name glm --max-model-len 16384 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --load-format instanttensor --no-enable-flashinfer-autotune --trust-remote-code \
  --limit-mm-per-prompt '{"image":0,"video":0}' --enable-auto-tool-choice --tool-call-parser glm47

Measured on one RTX PRO 6000 Blackwell (96 GB), converted checkpoint 68.5 GiB, KV cache ~6 GiB at 0.90 utilization. Tool calls come back as standard OpenAI tool_calls; without a reasoning parser the reasoning is returned inline before </think>.

Agent tests through vLLM (same two tasks as above, temperature 0, real tools, standard OpenAI tool-call history):

Notes: the converted checkpoint is 6.5 GiB larger than the published one (BF16 instead of FP8 for the dense parts) and the block-scale rounding in step 2 adds noise comparable to a few % on the affected scale tensors; no benchmark suite was run on the vLLM path.

Not yet verified

• Plain transformers from_pretrained — not tested.

Limitations

• The base model is abliterated (refusal-removed): outputs are unfiltered. You are responsible for how you use it.
• Expert pruning trades breadth for size: expect weaker world knowledge, science and rare languages.
• Evaluation above is teacher-forced perplexity plus two functional agent tasks — not a benchmark suite.

---

Кратко по-русски

Урезанная до 92 экспертов на слой версия GLM-5.3-Flash Uncensored NVFP4 (~107 млрд параметров, 62 ГиБ), целиком помещается в одну видеокарту на 96 ГБ. Эксперты отбирались под агентность, вызовы инструментов в родном формате GLM, код, русский и английский диалог; научные и энциклопедические знания сознательно не защищались. Проверено: perplexity на отложенных данных и агентные задачи с реальными инструментами (обе решены, без ошибок в вызовах). Запуск — движок из папки inference/ (OpenAI-совместимый API).