DeepSeek-V4-Flash-V100

DeepSeek-V4-Flash on V100 — working serving recipe, bug fix, and measured numbers

<!-- pli:blurb:start -->

Published by PLI Labs, the open research arm of Proprietary Legal Intelligence. PLI builds AI systems around a legal organization's own work product, combining retrieval, staged workflows, validation, and attorney review. Visit proprietarylegal.ai to explore our work and discuss what your organization needs.

<!-- pli:blurb:end -->

PLI Labs. This repository documents (and ships the patches for) running DeepSeek-V4-Flash (284B/~13B-active MoE) correctly on eight 2017-era NVIDIA V100-SXM2-32GB GPUs with llama.cpp — to our knowledge the first published Volta datapoint — including the root cause and fix for a silent-output-corruption bug that affects every backend when the K-cache is quantized (upstream: ggml-org/llama.cpp#25382).

No model weights here: use unsloth's GGUF quants (unsloth/DeepSeek-V4-Flash-GGUF, UD-Q4_K_XL, 144 GiB) and verify shard sha256s against the LFS manifest.

Measured (8× V100, layer split, FA on, power-capped 245 W/GPU)

The one thing you must know

With --cache-type-k q8_0, this model emits confident gibberish behind a healthy /health — on CPU and CUDA alike. Cause: llama.cpp's quantized-KV Hadamard rotation diverts DeepSeek-V4 off its sparse attention paths into a fallback with broken rotation math. Either run --cache-type-k f16 (zero-cost, MLA's cache is tiny) or apply the patches here (also on the ds4-volta-fix branch), which disable the rotation for this architecture — verified with a full K/V-type matrix on CPU and GPU.

Quickstart

git clone -b ds4-volta-fix https://github.com/Mermiges/llama.cpp
cmake -S llama.cpp -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j
build/bin/llama-server -m DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf \
  -ngl 999 -sm layer -fa 1 -b 2048 -ub 1024 -c 8192 \
  --cache-type-k f16 --cache-type-v f16

Update: the proper fix (rotation-aware attention) is also available

Beyond the minimal fix (patches in the repo root), patches-rotation-aware/ (branch ds4-rotation-aware) makes DeepSeek-V4's attention paths rotation-aware and re-enables the quantized-KV incoherence rotation — verified coherent across the full K/V-type matrix, ~6.8% decode cost, and in a ~6000-token needle screen the rotation-ON build retrieved a needle that plain quantized-K missed. patches-tensor-parallel/ carries the experimental de-blacklisting of DeepSeek-V4 for llama.cpp's tensor-parallel mode (loads and reaches decode on 8× V100; one scheduler assert remains).

Full research record (evidence matrices, raw benchmarks, negative results): https://github.com/ProprietaryLegal/deepseek-v4-flash-v100

PLI Labs — Proprietary Legal Intelligence, LLC. Research artifacts for lawyer-supervised, on-premises legal AI; nothing here is legal advice.

<!-- pli:footer:start -->

---

About PLI Labs

“Your proprietary work product is your proprietary intelligence.” PLI Labs is the open research arm of Proprietary Legal Intelligence. We publish legal model prunes, serving recipes for owned hardware, and open-source tools so readers can inspect our methods and test our claims before hiring us.

Founded by a practicing trial lawyer, PLI builds AI systems around a legal organization's own work product: retrieval, staged workflows, validation, and attorney review. The approach grew from systems running on live matters in a litigation practice. Our consulting work supports deployment through approved frontier APIs, private cloud, or open-weight models on hardware you own. Explore the research or contact us about your organization's work.

Website · PLI Labs · Research · GitHub · Hugging Face · Contact

Proprietary Legal Intelligence LLC is a technology consultancy, not a law firm, and does not provide legal services or legal advice.

<!-- pli:footer:end -->