DeepSeek-V4-Pro-671B-GALE-1bit-v4

DeepSeek-V4-Pro — GALE 1-bit experts (v4, error-feedback)

A 1-bit re-quantization of DeepSeek-V4-Pro's routed experts, plus a server that auto-detects its hardware: it was built and measured on two RTX 3090s, one RTX 2080 Ti and a Ryzen 9 3900X, but the device planning is generic (see Running it) — from a single GPU up to a multi-GPU node big enough to hold every expert resident, with no config to hand-tune per machine.

What is in the pack

End-to-end: ~1.39 bits/weight in VRAM (1.46 on disk) over ~1.57 T parameters counted from the tensor shapes.

How the experts were quantized (v4)

The v3 pack rounded each weight to its sign. v4 re-chose every sign with GPTQ-style error feedback in output space: for each expert matrix, the rounding error of each column is pushed into the columns not yet quantized, weighted by the inverse Hessian of real activations (16k tokens captured from the model itself, ridge 0.3), and a least-squares per-row gain is folded into the block scales at zero extra bits. Same format and size as v3.

Measured on held-out activations (expert output, relative error): naive sign rounding 0.74 → v4 0.52–0.54 (in-sample 0.38–0.52 by layer). The model answers short questions cleanly and writes small functions; coherence still degrades over long answers, so the server ends a turn as soon as the model finishes one (see Ending a turn) rather than letting it run to a token cap.

The 4-bit non-expert weights are symmetric int4 per 128-block with fp16 scales, 0.119 relative error on this model's weights (a Hadamard rotation was measured and not worth it).

Running it

git clone / download this repo, cd into it
pip install "torch>=2.5" triton transformers tokenizers fastapi uvicorn
ulimit -n 65536
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
python3 server/deepseek_v4pro_server_v2.py

Each of those lines is there because its absence broke the first run on a machine that wasn't the one it was built on (a 4×A100-80GB rental, 2026-09-03): torch 2.4 lacks torch.float8_e8m0fnu, which the reference model uses; the pack is ~250 files plus per-expert reads, so the default 1024 open-file limit dies mid-generation; and full residency fills each GPU to ~90%, where the default allocator fragments (9.7 GiB "reserved but unallocated") and OOMs. server/tilelang/ is a stub: the reference kernel.py imports tilelang unconditionally, but the server replaces every tilelang kernel with plain torch before any is called, so the real package (and the Hopper-class hardware it targets) is not required.

Knobs, all environment variables. Serving features added 2026-09-06 (all on by default):

• Prefix caching / KV reuse (DSV4_KV_REUSE=0 to disable). The unchanging preamble of a conversation is prefilled once, snapshotted, and persisted under server/kv_snapshots/ (DSV4_KV_SNAPSHOT_DIR, DSV4_KV_SNAPSHOTS=8); a rolling checkpoint is taken every DSV4_KV_CKPT_EVERY=8 decode tokens (DSV4_KV_CKPT_KEEP=6). A request that extends any cached state re-uses it and feeds only the new tail. Measured on the tiered box: a 1230-token agent prompt went 464 s → 6 s, and 0 s when the whole prompt is already cached. Because the reference implementation's batched path only supports position 0, the tail is fed one token at a time; DSV4_KV_INCR_MAX=400 is the point where a full batched prefill wins instead.
• Ending a turn. This 1-bit model does not reliably emit EOS. DSV4_ONESHOT puts one plain-text example turn in the preamble (that alone took a greeting from 512 tokens to 9), DSV4_STOP_TURN=1 ends the turn when the model starts writing the next User: turn itself, and DSV4_LOOP_STOP=8 trims a trailing repeated n-gram. An example ending in the model's real EOS token produced junk, which is why the default is plain text.
• Residency-aware routing (DSV4_TIER_BIAS="0.15,0.30", empty to disable, tiered mode only). Adds a per-expert penalty to each gate's selection bias — DeepSeek's aux-loss-free balancing term, which affects top-k selection only and never the routing weights — so near-tie picks land on experts that are already resident. Measured 0.36–0.65 → 1.2 tok/s with the CPU-tier share falling 62% → 17%. Do not raise it blindly: 0.30/0.60 was faster still (~2 tok/s) but visibly damaged instruction-following ("answer with just the number" became a 74-token ramble). DSV4_LINK_AWARE_BIAS=1 measures each device's real round-trip latency at startup and scales its penalty (it found a PCIe-x1-downtrained card at 0.10 ms vs 0.03 ms and doubled that card's penalty automatically).
• Workload re-pinning. DSV4_COUNT_ROUTING=<path> accumulates real routing counts while serving and re-seeds them at startup; point DSV4_ROUTING_COUNTS at a blend of those with the shipped routing_counts.json to pin the hot set for your workload. On a coding workload the top-96 experts per layer overlapped the shipped calibration ranking by only ~50–70%.
• Expert compute (DSV4_DIRECT_GEMV: 0 = unpack then matmul, 1 = direct, default; 2 = batched). dsv4_kernels.packed_gemv multiplies straight out of a packed record (signs, block scales and int4 outliers decoded in registers), 5.6–6.2× faster per matrix than unpacking to bf16 first and closer to the fp32 oracle. DSV4_FWHT=butterfly uses a single-kernel Walsh-Hadamard transform (3.3–7.0× the matmul form, exact to 1e-8). Note both were neutral on end-to-end tok/s in tiered mode, where each layer waits on the CPU tier — they are headroom for configurations where the GPU path is the critical path.
• DSV4_PROFILE=1 prints a per-token breakdown, but crashes a ~1200-token batched prefill with CUBLAS_STATUS_INTERNAL_ERROR on the tiered box; use it for short prompts only.

DSV4_RESERVE — fraction of each GPU's free memory the planner may fill with weights (default 0.85; 4×80GB needs 0.88 to go fully resident — 16 layers of experts is 67.5 GB). DSV4_PROSE_STOPS=0 — turns off the chat-tuned stop heuristics (junk detector, soft stop at a sentence end, \n\n\n) that would otherwise cut code at the eighth bracket; use it for benchmarks or code. /v1/chat/completions also honours OpenAI-style stop sequences (string or list). DSV4_HARD_CAP / DSV4_SOFT_STOP — output caps for chat (default 64 / 24 tokens; raise for code).

Nothing to edit for a different machine — Engine.__init__ enumerates every visible GPU (torch.cuda.device_count()), reads each one's free memory, and picks one of two plans:

• Full residency — if every expert fits resident across the visible GPUs (with a safety margin for KV cache/activations), layers are split across all of them proportional to free memory, every expert is pinned once at startup, and the CPU/overflow tiers are skipped entirely — there's nothing for them to do. This is the plan on a big enough rental (e.g. a multi-GPU 80GB-class node).
• Tiered — if it doesn't fit (this project's original 2×24GB 3090s + 1×11GB 2080 Ti), layers split across the biggest devices proportional to free memory; each holds a hot-expert cache sized independently (DSV4_VRAM_CACHE_GB) rather than a full layer's worth; a spare device (if one exists beyond what's needed to hold layers) becomes a second, hit-only expert cache; and an AVX2 kernel computes whatever's resident nowhere from system RAM. This is the exact configuration the numbers below were measured on.

Paths are derived from DSV4_ROOT (default: this repo's own root, so cloning and running just works) — override DSV4_PACKED_DIR, DSV4_MISC_DIR, DSV4_MANIFEST, DSV4_ROUTING_COUNTS, DSV4_PREDICTOR individually if you lay files out differently. server/ bundles everything needed to run standalone: the official reference implementation (server/reference/, from deepseek-ai/DeepSeek-V4-Pro), the tokenizer, the CPU kernel source and compiled .so, the quantizer, and calibration artifacts (routing counts, expert predictor).

It exposes OpenAI-style /v1/chat/completions and Ollama-style /api/chat, text only. Tool calling lives in a separate bridge, server/agent_bridge/ — run it instead of the server directly, from the repo root: python3 server/agent_bridge/agent_server.py --runtime . --weights . (it imports the server as a library and serves on port 11443). it renders OpenAI-style tools into DeepSeek's DSML wire format, validates the model's calls against each tool's JSON Schema, and never executes anything itself — the client owns execution and permissions. Needs jsonschema. Two caveats measured 2026-09-06: the model reliably emits a well-formed call and then narrates fabricated results, so the turn ends at the first closed tool block and anything after it is dropped; and the tool schemas themselves cost ~1100 prompt tokens against a 4096-token context, which is exactly what the prefix cache above is for. Kernels are Triton (any CUDA GPU) and x86 AVX2 (server/dsv4_cpu_expert.c, used only in tiered mode); nothing runs on Apple Silicon or ARM as-is.

Measured, both configurations:

The tiered column is the same hardware as the original measurement plus a second 2080 Ti, with the 2026-09-06 serving work above. The gain is from routing and caching, not kernels: an end-to-end agent greeting went from ~17 minutes to 4 seconds. Weights moved from an external USB drive to NVMe part way through, which is worth ~120× on cold prefill by itself — don't serve this pack off USB.

Started the day at 0.23 tok/s; the profile-driven changes are in the commit history of the server file. The honest reading of the A100 column: taking PCIe, RAM bandwidth and CPU compute out of the picture entirely bought 1.6×, not 10× — at batch 1 the floor is the ~366 expert unpacks per token plus 61 layers × ~200 small kernel launches, i.e. launch overhead and the unpack kernel, not data movement. CUDA graphs for the dense per-layer part and a batched multi-expert unpack are the next levers, and they would help both configurations.

Files

• packed_v4/ — expert pack (layer_LLL.hP.pack, 4 part files per layer) + manifest_mixed.jsonl (every tensor: kind, shape, sections, pack file, byte offset — resolved by filename relative to wherever you put packed_v4/, not by any path baked in at build time).
• misc_v3/ — non-expert tensors (misc_layer_LLL.pack, fp8/fp16), referenced by the same manifest.
• server/ — everything needed to run standalone: the server (deepseek_v4pro_server_v2.py), the official reference implementation (reference/model.py, reference/kernel.py, from deepseek-ai/DeepSeek-V4-Pro), the tokenizer, kernels (dsv4_kernels.py, dsv4_w4.py, dsv4_cpu_expert.c + compiled .so, dsv4_cpu_tier.py), the quantizer (v4pro_gptq1bit.py), calibration artifacts (routing_counts.json, expert_predictor.pt + its fit script), an example systemd unit, the chat CLI, and agent_bridge/ (the tool-calling front end: agent_server.py, tool_protocol.py).

Honest limits

• Quality: short answers. It will drive a coding agent end to end — with the bridge above it wrote a file and ran it through bash on request — but not reliably: it fabricates tool results if allowed to keep going, and on a 4-prompt probe it answered arithmetic and a shell-command question correctly while writing prose instead of a Fibonacci function. Next levers: re-calibrate on code, then fine-tune scales/outliers layer-wise (the sign bits cannot be trained on any single machine — that needs the full model plus gradients in memory).
• For an agent specifically, the binding cost is now feeding tool output back in: ~0.7 s per token of tool result in tiered mode, because appended context is fed one token at a time (see DSV4_KV_INCR_MAX). Making the reference model's batched path work at non-zero position is the highest-value change left for agent use.
• Speed ceiling in the tiered configuration is still roughly 2 tok/s even with residency-aware routing; the model is 257 GB of experts against ~27 GB of resident VRAM on that hardware, and every token touches ~4 GB of them. A node with enough combined VRAM to hold everything resident removes that bottleneck entirely, but hasn't been measured — see Running it.
• Prefix caching exists as of 2026-09-06 (see Running it), and snapshots are matched by prompt text, so any change to the system prompt — including one that embeds a working directory — starts a new snapshot and pays one full prefill.
• The expert files were built from the official deepseek-ai/DeepSeek-V4-Pro weights streamed shard by shard; the original weights are not included here.
• Kernels are Triton (any CUDA GPU) and x86 AVX2 (the CPU tier, tiered mode only); nothing runs on Apple Silicon or ARM as-is.