DeepSeek-V4.1-Flash-NVFP4

DeepSeek-V4.1-Flash-NVFP4

475.2 GiB → 399.9 GiB (−15.8%), with the routed experts converted to NVFP4 bit-exactly and the Engram memory tables taken from FP8 to FP4.

An FP4 re-pack of deepseek-ai/DeepSeek-V4.1-Flash by LibertAI.

⚠️ Read this first: V4.1-Flash already ships in 4-bit

If you came here expecting the usual ~70% NVFP4 cut, it does not exist for this model. DeepSeek quantized V4.1-Flash themselves. From the upstream config.json:

"quantization_config": {
  "quant_method": "fp8", "weight_block_size": [32, 32],
  "scale_fmt": "ue8m0", "expert_dtype": "fp4"
}

Measured across all 48 upstream shards / 96,085 tensors (475.2 GiB):

This is the break from V4-Flash. DeepSeek-V4-Flash-0731 shipped FP8 block-128×128 with no FP4 anywhere — which is why the NVFP4 conversions of that model were worth ~2×. Here the 57% of the checkpoint that is routed experts is already at 4 bits, so quantizing it again buys nothing. The Engram tables are the only large block left at 8 bits, and they are what this repo is really about.

What this repo does

1. Experts: MXFP4 → NVFP4, bit-exactly

Upstream experts are MX-style: 32-element blocks with a power-of-two E8M0 scale. NVFP4 is 16-element blocks with an E4M3 scale plus an FP32 per-tensor global scale.

The transcode is lossless, for a structural reason: NVFP4's 16-element blocks are a strict refinement of the source's 32-element blocks, so both NVFP4 half-blocks inherit one source scale s. Emit the nibbles unchanged and force weight_scale × weight_scale_2 == s, and every reconstructed value is identical. E8M0 scales are powers of two, and E4M3 represents 2^k exactly for k ∈ [−9, 8], so the identity holds for every block whose scale lands in that window. The global scale is chosen per tensor to centre it.

Measured over the whole checkpoint:

This costs size. NVFP4 is 4 + 8/16 = 4.5 bits/weight against MXFP4's 4 + 8/32 = 4.25, so the experts grow — one shard went 7,389,759,032 → 7,814,559,376 bytes, +5.75%. Take the expert half only if your engine's Blackwell MoE kernels want NVFP4 and have no MX path.

Prior art: this losslessness is not something we invented, it is what any careful implementation gets. We checked s-zaizen/DeepSeek-V4.1-Flash-NVFP4, published the same day, by reconstructing layers.1.ffn.experts.0.w1 from their checkpoint and from the source: their nibbles are identical and all 11,796,480 elements match exactly. Their repo is 491.1 GiB — larger than the 475.2 GiB original — because the expert transcode grows the file and Engram is left at FP8.

2. Engram: FP8 → FP4 — the actual size lever

The Engram conditional-memory tables are two tensors of ~384M rows × 256, 94.6 GiB each. We take them to E2M1 keeping the source's own block-32 / E8M0 scale layout, so a table row is stored exactly the way DeepSeek stores an expert weight and any kernel that dequantizes an expert can dequantize the table. Scales are recomputed as a power of two ≥ amax/6 rather than reused, so no block clips.

⚠️ This half is lossy and has not been evaluated end-to-end. Cosine ≈ 0.9934 is lower than we would accept on expert weights, and no benchmark has been run — no engine can run this architecture yet (see below). If you need a checkpoint with untouched Engram, use upstream or s-zaizen's repo. We publish this because the size and I/O win is large and the measurement is stated honestly, not because the quality is established.

Net effect

143,317 tensors across 48 shards. The index is rebuilt from the shard headers, and every tensor the source had is present.

Running Engram off SSD

Engram is a hashed n-gram gather, not a GEMM. With engram_layer_ids [1,14], engram_max_ngram_size 4 and engram_n_heads 8, each position looks up 3 n-gram sizes × 8 heads = 24 rows per Engram layer, so 48 rows per token:

At decode that is ~48 random reads per token — trivial for an NVMe. Prefill is where it bites: a 64K-token prompt is ~3.1M small random reads. So the FP4 Engram halves the SSD traffic per token as well as the footprint. Upstream anticipates this: their convert.py already shards engram.embed along dim 0 across ranks.

But it does not make small boxes work. Everything except the two Engram tables is 302.3 GiB (324.6 GB) and must be resident. Two 120 GB unified-memory boxes cannot hold it, with or without Engram on disk. A 4×B200 or an 8×96 GB node can.

On pruning

We looked at whether V4.1-Flash could be expert-pruned to fit smaller hardware without calibration data. It cannot, and the checkpoint says so three ways:

• the noaux_tc router bias is nearly uniform — std 0.0627 against absolute values of 9–57, top/bottom-decile gap ~0.13. Load balancing has equalised expert usage, so there are no cold experts to drop.
• router directions are not redundant — nearest-neighbour cosine averages 0.32–0.45, and 0–2 experts per layer exceed 0.9. Nothing to merge.
• ⚠️ gate.bias and gate.bias_vl are effectively uncorrelated (mean −0.05, min −0.45, mean absolute difference 6.4). The model carries a separate router bias for vision-language tokens, so expert specialisation is modality-dependent and a text-only REAP calibration would prune the vision path. If you are pruning this model, calibrate on multimodal data.

Any real prune needs activation-based REAP, which needs a working engine.

Engine support

As of 2026-09-10 no released engine serves deepseek_v41. vLLM has open PRs (#56214, #56228, #56227, #56208) filed the morning the model dropped; SGLang has merged a cookbook plus initial support.

⚠️ DeepSeek's own inference/ stack is bundled here for reference but will NOT load this checkpoint as-is. It expects the upstream layout: convert.py asserts routed experts are torch.int8 and reads their scales from <name>.scale, whereas this repo stores them as uint8 with <name>.weight_scale (E4M3, block 16) plus a <name>.weight_scale_2 global scale, and stores the Engram tables at E2M1 rather than E4M3. Loading it needs a reader that understands the quantization_config below. We ship the reference code unmodified so the delta is inspectable, not because it works out of the box.

Because nothing reads this architecture yet, the quantization_config in config.json describes what we did rather than matching an existing convention:

"expert_dtype": "nvfp4", "expert_block_size": 16, "expert_scale_fmt": "e4m3",
"expert_global_scale": true,
"engram_dtype": "fp4", "engram_block_size": 32, "engram_scale_fmt": "ue8m0"

Dense/attention weights are untouched (fp8, block 32×32, ue8m0). No throughput, latency or quality numbers are published here, because we have not been able to run the model.

About the base model

DeepSeek-V4.1-Flash is a multimodal MoE with a 552B-parameter backbone plus 196B of sparsely-accessed Engram memory, activating 8B parameters per token during prefill and 16B during decode, with a 1M-token context.

It uses a Causal Encoder-Decoder layout (40 layers = a 20-layer causal encoder feeding a 20-layer decoder), Compressed Sparse Attention 2 with per-layer Full/Reindex/Reuse modes and a hierarchical sparse indexer, FP4 main KV caching for a global KV footprint of 890 bytes/token, Single-Pass mHC, and DSpark speculative decoding. 384 routed experts

• 1 shared, 6 routed per token. Reasoning effort is continuously controllable from 1 to 100.

See the technical report for evaluations. We have run none of our own.

Provenance

Built with dsv41_fp4_stream.py, a shard-streaming re-packer. The E2M1 table and nibble order are taken verbatim from the checkpoint's own inference/convert.py, not reconstructed from the spec. The Engram shards are written as hand-built safetensors containers so a 94.6 GiB table is never held in RAM.

Also included: verify_transcode.py (the bit-exactness check), test_engram_path.py and test_engram_shard.py (cpu/cuda agreement and container round-trip), and analyze_router.py (the pruning analysis above).

License: MIT, inherited from the base model.