DeepSeek-V4.1-Flash-EXL3-2.9bpw

DeepSeek-V4.1-Flash · EXL3 · 2.9bpw

EXL3 quantization of deepseek-ai/DeepSeek-V4.1-Flash, converted with exllamav3 v1.4.2.

This is not a drop-in replacement for the original FP8 checkpoint. The Engram embedding tables are not in this repo — you still need shards 47 and 48 from the base model (~95 GiB each).

A slightly denser sibling lives at Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw.

Engram tables (required)

EXL3 only quantized the Engram wkv linears. The FP8 row tables stay in the original DeepSeek tree:

• model-00047-of-00048.safetensors (~95 GiB)
• model-00048-of-00048.safetensors (~95 GiB)

Point the runtime at that directory. This card ships engram_table_dir as null on purpose — do not leave a machine-local path in config.json.

vLLM:

--quantization exl3 \
--hf-overrides '{"engram_table_dir":"/path/to/DeepSeek-V4.1-Flash"}'

A slimmer overlay is enough: hardlink or copy only shards 47–48 (plus an embed-only index if your stack wants one). You do not need the other 46 original shards once this EXL3 tree is loaded.

Download

hf download Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw \
  --local-dir ./DeepSeek-V4.1-Flash-EXL3-2.9bpw

hf download deepseek-ai/DeepSeek-V4.1-Flash \
  --include "model-00047-of-00048.safetensors" \
  --include "model-00048-of-00048.safetensors" \
  --local-dir ./DeepSeek-V4.1-Flash

Runtime

This checkpoint is meant for stacks that load EXL3 (trellis / suh / svh / mul1), not vanilla Transformers weight loaders.

• vLLM with EXL3 + DeepSeek-V4.1 support (--quantization exl3, DeepSeek V4.1 tokenizer / tool / reasoning parsers). Native DSpark: --speculative-config '{"method":"dspark","num_speculative_tokens":3}'.
• ExLlamaV3 / TabbyAPI once the architecture is supported in the build you are running.

Official sampling from the base card:

Thinking defaults on. Disable per request with chat-template kwargs (enable_thinking=false) when you want a non-reasoning smoke.

KV cache is native V4.1 FP4 (~890 B/token). Do not force an unrelated fp8 KV dtype from other recipes.

Conversion

python convert.py \
  -i DeepSeek-V4.1-Flash \
  -o DeepSeek-V4.1-Flash-EXL3-2.9bpw \
  -w /path/to/work \
  -b 2.9 \
  -hb 6 \
  -hq \
  --codebook mul1 \
  -cd cal_trace_dsv41_flash_workload.json

Final reported decoder bitrate: 2.90 bpw (--hq enabled). MTP tensors were quantized at 4 bpw. Routed experts are a 2/3 mix (mostly K=3, a slice at K=2); attention / shared / Engram wkv sit higher (typically 4–6). Indexer wk is 8 bpw. Embeddings stay bf16.

Base model

DeepSeek-V4.1-Flash is a multimodal MoE (552B backbone, 8B/16B activated per token in prefill/decode) with CED, CSA2, Engram memory, and in-checkpoint DSpark. Architecture, evals, and prompt format live on the original model card and tech report. Scores there are for the unquantized model, not this EXL3 build.

License

MIT, inherited from deepseek-ai/DeepSeek-V4.1-Flash. Quantization adds no extra restrictions.

Citation

@misc{deepseekai2026deepseekv41flash,
      title={DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression},
      author={DeepSeek-AI},
      year={2026},
}

Quantized by Mia's AI Lab. Format and tooling: exllamav3 (turboderp).