DeepSeek-V4-Flash-exl3

DeepSeek-V4-Flash — ExLlamaV3 (exl3) quant · EXPERIMENTAL

ExLlamaV3 (exl3) quantizations of DeepSeek-V4-Flash.

[!WARNING] This quant does NOT run on stock exllamav3 or stock tabbyAPI. DeepSeek-V4-Flash uses an architecture that upstream exllamav3 does not implement, so loading these files with an official release will fail with an unknown-architecture error. You must install the exllamav3 fork branch described in Installation. This is experimental software — expect rough edges and breaking changes.

---

Available quants

Weights live on a branch named after the bit-rate. The model card (this page) is on main.

More bit-rates may be published as additional branches.

---

Why it isn't supported natively in exllamav3

DeepSeek-V4-Flash is not a standard transformer, and none of the machinery it needs exists in any official exllamav3 release:

1. No DeepseekV4ForCausalLM architecture. Upstream has no model definition for it, so the loader rejects the checkpoint outright.
2. Custom compressed-attention CUDA kernels. The sliding-window + compressor + indexer attention is implemented in bespoke kernels (dsv4.cu) that ship only on the fork.
3. QAT fake-quant path. The model was trained with quantization-aware training; reproducing its fp8/fp4 fake-quants at inference (for output fidelity) requires extra code paths upstream does not have.
4. Non-standard cache model. The compressor and sliding-window state is stateful and kept inside the attention module, not laid out like exllamav3's normal paged KV cache. The fork adds two bespoke paths to make caching work anyway — append-only prefix reuse (default) and an optional native-paging mode for multiple concurrent sessions (see Cache reuse & multi-session) — neither of which upstream carries.
5. Custom MoE router and loader/generator hooks that aren't part of the stock modules.

All of this lives on the fork branch below.

---

Installation

Three steps: install the tabbyAPI fork, install the exllamav3 fork into it, download the model. You'll need an NVIDIA CUDA toolkit (the exllamav3 fork builds a custom extension) and enough VRAM (4.0bpw weights are ~145 GB).

1. Install the tabbyAPI fork

The fork branch adds the DeepSeek-V4 tool-call parser and an example config on top of current tabbyAPI. Clone it and set up its venv:

git clone -b deepseek-v4-flash https://github.com/NeuroSenko/tabbyAPI.git
cd tabbyAPI
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install .

2. Install the exllamav3 fork into that venv

That last step pulled stock exllamav3, which has no DeepSeek-V4 support. Replace it with the fork branch (tabby's venv still active):

pip install --force-reinstall --no-deps --no-build-isolation \
  "git+https://github.com/NeuroSenko/exllamav3.git@deepseek-v4-flash"

This compiles the custom CUDA extension (needs the CUDA toolkit) and cleans up after itself. The fork is a drop-in superset of exllamav3 v1.0.0, so the rest of tabby keeps working.

3. Download the model

The weights are on the 4.0bpw branch, not main — select that revision:

huggingface-cli download NeuroSenko/DeepSeek-V4-Flash-exl3 --revision 4.0bpw \
  --local-dir ./DeepSeek-V4-Flash-exl3-4.0bpw

Run

Point a config at the folder and start tabby from the venv:

model:
  model_dir: /path/to/models
  model_name: DeepSeek-V4-Flash-exl3-4.0bpw
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262400
  cache_mode: FP16
  gpu_split_auto: true
  tool_format: deepseek_v4

python main.py --config config_dsv4.yml

[!TIP] If autosplit OOMs while loading across multiple GPUs, give each GPU a few GB of headroom with a per-GPU autosplit_reserve list — one entry per GPU, e.g. [3072, 3072, 3072, 3072]. A single-element list only reserves GPU0. In paged mode the cache is larger (see below), so you may need to lower the reserve instead so it fits.

---

Cache reuse & multi-session

Because the compressed attention state lives inside the module rather than in a standard paged KV pool, the fork provides two ways to avoid re-prefilling a prompt from scratch. Both decode at the same speed — reuse only affects prefill / time-to-first-token.

Default — append-only prefix reuse. The cache stays in its compact non-paged layout (~3.6 GB at 256k) and a single, evolving conversation continues from the previous turn's cache instead of replaying the whole prompt each turn. This is the common agent / tool-loop case: a long context that grows by a few tokens per turn keeps TTFT low. Only the one resident sequence's prefix is reusable — switching to a different conversation re-prefills.

DSV4_PAGED=1 — native paging + multi-session. Materializes the per-page cache state so exllamav3's own page table can share and reuse pages by content hash. Several independent conversations can stay resident at once, and switching between them reuses each one's cache without re-prefilling — even after other sessions were served in between. The trade-off is memory: paging stores per-page state, so the cache is roughly 3× larger (~11 GB at 256k) and must fit alongside the weights (lower autosplit_reserve if the split doesn't fit).

Both are set via environment variables when launching tabby: DSV4_PAGED (default 0) toggles paging; DSV4_REUSE (default 1) toggles the append-only reuse in the non-paged mode.

DSV4_PAGED=1 python main.py --config config_dsv4.yml

---

How the fork works (internals)

For the curious, this is what the fork adds on top of exllamav3:

• Compressed sparse attention. Each layer uses one of three KV regimes: a 128-token sliding window; CSA — a 4:1 compressor plus a top-512 indexer that selects the most relevant compressed blocks; or HCA — a 128:1 compressor only. Only a small, selected slice of the past is ever attended to, which is why the KV cache stays around 3.6 GB even at 256k tokens.
• Latent attention (MQA, not MLA). Keys and values are a single shared 512-dim latent (num_key_value_heads == 1), used directly — there is no per-head KV up-projection, so this is MQA rather than MLA. Queries and the attention output use low-rank projections, and RoPE is decoupled and scaled with YaRN for long context.
• Mixture-of-Experts. 256 fine-grained routed experts with sigmoid gating and routed-scaling, 6 active per token, plus 1 always-on shared expert.
• Fidelity-preserving fake-quant. The model was trained with quantization-aware training. The fork reproduces the reference fake-quants at inference — fp8 on the attention KV "nope" dims, fp4 for the indexer/compressor, and the SwiGLU activation clamp — so the quant matches how the model was trained. Weights use exl3's trellis quantization at the target bit-rate; activations stay in fp16 (higher precision than the reference's fp8 activation path, which is harmless).
• Custom kernels + engine hooks. New CUDA kernels implement the compressed attention, with hooks into the loader, cache, generator (chunked page-aligned prefill, append-only prefix reuse, and an optional native-paging cache layer for multi-session reuse) and the MoE path.

---

Limitations

• Experimental / not upstream. Branches and behavior may change.
• Cache reuse has two modes. By default a single evolving conversation reuses its own growing prefix (append-only), so a multi-turn chat / tool loop continues from the previous turn instead of re-prefilling. Cross-session reuse — keeping several conversations resident and switching between them — requires paged mode (DSV4_PAGED=1), which costs more cache memory. See Cache reuse & multi-session.
• One sequence per forward pass. Tested with max_batch_size: 1. Paged mode lets multiple sessions share the cache and be served in turn, but concurrent batched decoding of several sequences at once is untested.
• Build required. A CUDA toolkit is needed to compile the custom extension.
• Large. 4.0bpw is ~145 GB of weights — you need substantial multi-GPU VRAM.

---

Credits & links

• Base model: DeepSeek-V4-Flash by DeepSeek.
• exllamav3 by turboderp — https://github.com/turboderp-org/exllamav3
• Engine fork (support branch): https://github.com/NeuroSenko/exllamav3/tree/deepseek-v4-flash
• tabbyAPI fork (tool calling): https://github.com/NeuroSenko/tabbyAPI/tree/deepseek-v4-flash