Downloads ยท 30 days
790
18% of all-time downloads
Kevletesteur/DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF
DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF is a machine learning model from Kevletesteur. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
TL;DR ๐ฏ A 284B MoE in 103 GB (2.90 bpw) that we verified against the full-precision official API on identical protocols: ๐ข 90.8% token-identical, ๐ข 240/240 paired-QA parity, ๐ข deep-derivation parity (SymPy-checkedโฆ
Downloads ยท 30 days
790
18% of all-time downloads
All-time downloads
4.5K
Public
Repo size
103 GB
Likes
19
Trending 3
Click a slice to open those files.
.gguf103 GB ยท 100%
From the Hugging Face model README
TL;DR ๐ฏ A 284B MoE in 103 GB (2.90 bpw) that we verified against the full-precision official API on identical protocols: ๐ข 90.8% token-identical, ๐ข 240/240 paired-QA parity, ๐ข deep-derivation parity (SymPy-checked), ๐ข 0.0% confident-wrong-answer rate. Tuned end-to-end for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, 115 GB unified RAM) on mainline llama.cpp โ no fork. Every flag below is a measurement, not a vibe. ๐
What's different from the original unsloth UD-IQ3_XXS? The three attention tensor
families that dominate the per-token byte budget (attn_q_b, attn_output_a,
attn_output_b โ 51.9% of all bytes read per token, more than the 256 experts combined!)
are requantized Q8_0 โ Q6_K from the clean source. Measured effect: +23.8% generation
on long-form output, and draft acceptance jumping 44% โ 63% (less attention noise keeps
the speculative drafter and the target in agreement). ๐
# 1๏ธโฃ One-time: raise the unified-memory GPU ceiling (the single biggest win: ร2.8 !)
# Default GTT = half your RAM. On a 128 GB box you want ~110 GiB:
echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT amdgpu.gttsize=112640 ttm.pages_limit=28835840"' \
| sudo tee /etc/default/grub.d/99-amdgpu-gtt.cfg
sudo update-grub && sudo reboot
# 2๏ธโฃ Build llama.cpp (mainline!) for ROCm/HIP
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
# 3๏ธโฃ Download this repo (model + the DSpark drafter from unsloth's repo)
hf download Kevletesteur/DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf
# โ requantize the drafter to Q3_K_S (8.5 GB) with llama-quantize, or use it as-is (10.9 GB)
# โ
Verify integrity BEFORE first launch (we learned this the hard way):
sha256sum -c <(echo "538ec6932c50c6993f8a2cb18dc138e0fb5f6a7c2659e53a7566f1dc8ebb4344 DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf")
GGML_CUDA_DISABLE_GRAPHS=1 LLAMA_NO_FUSE_HC_POST=1 \
./build/bin/llama-server \
-m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf \
--no-mmap -fa on -ctk q8_0 -ctv q8_0 \
-ngl 99 -t 12 -b 2048 -ub 512 -c 262144 \
--n-cpu-moe 16 \
--model-draft DSpark-draft-Q3_K_S.gguf --spec-type draft-dspark \
--spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
--temp 1.0 --top-p 0.95 --min-p 0.0
| flag | why (measured) |
|---|---|
--no-mmap | ๐ด mandatory on unified memory โ with mmap the model exists twice (GPU buffer + page cache): 51 min of thrashing vs 40 s load |
GGML_CUDA_DISABLE_GRAPHS=1 | +3.0% โ a 256-expert MoE re-routes every token; HIP graph re-instantiation costs more than launch amortization saves |
LLAMA_NO_FUSE_HC_POST=1 | +1.5% โ this specific micro-fusion loses on gfx1151 |
-ctk/-ctv q8_0 | on THIS quant, f16 KV is โ20.6% (on the previous quant it was +21%โฆ settings do not transpose โ re-measure in your config) |
--n-cpu-moe 16 | frees 33 GB of GTT for 0.1% cost โ and prefill improves 2.4ร (the constraint is memory pressure, not compute) |
--spec-draft-n-max 2 | โ ๏ธ the optimum depends on context depth: 3 wins at short context, 2 wins from ~16k on (+3-4%). Long-context agents โ 2. There is a hard cliff at nโฅ4 (RDNA3 MMVQ max_batch=4 for these expert types) |
--temp 1.0 --top-p 0.95 | the values sourced from DeepSeek + unsloth (lower temp is faster via draft acceptance, but unsourced) |
| speed you should see | ~20-27 t/s short ctx (task-dependent!), ~14-16 t/s @16k, ~11 t/s @150k ยท prefill ~130 t/s |
/completion. The auto-injected <think> can
close instantly at temp 0 โ the model answers reflexively with 10-char reasoning traces
and looks broken. We "discovered" a factual-recall deficit that was 100% this artifact.
Use /v1/chat/completions for anything quality-related.draft-dspark with ngram modes. Syntactically accepted, destroys
DSpark (+101% โ +13% on tool calls). Ngram modes alone lose up to โ46%.mem_info_gtt_used, don't sleep 5.--slot-save-path + the save/restore API around restarts: 25 ms instead of
~6 min of re-prefill for a 30k-token session (ร14,000).Paired protocols, identical serving path both sides, instruments validated on synthetic cases before any model run (they caught 9 real bugs โ see the harness repo):
| probe | result vs full-precision API |
|---|---|
| ๐งฌ token-level teacher forcing (17,929 positions, 4 domains ร EN/FR) | 90.8% top-1 agreement ยท original's token outside local top-20: 0.04% |
| ๐งฎ 240-item exact-answer bank (McNemar, 6 families) | 240/240 = 240/240, zero discordance (gap bounded โฒ1.2%) |
| ๐ 60 multi-step derivations, SymPy oracle | parity |
| ๐ ๏ธ structured tool-call sequences | 25/25 both sides |
| ๐จ confident-wrong-answers P(answers โง wrong) | 0.0% |
| ๐ซ๐ท French | more token-faithful than English (88.1% vs 83.0% on physics) |
Also measured: Q5_K attention has better wikitext perplexity and worse real behavior (failed temp-0 probes, one repetition-loop) โ our third demonstration that PPL โ quality.
recipe/requant_attention.sh + recipe/tensor_types_q6kattn.txt rebuild this file from
unsloth's UD-IQ3_XXS shards in ~15 min (129 tensors requantized, 1,199 copied bit-identical,
every tensor explicitly pinned โ unpinned tensors silently fall into mixture heuristics).
๐ Kevletesteur/quant-vs-api-parity-harness โ
the complete benchmark toolkit (gates, teacher forcing, SymPy oracles, calibration), plus
NEGATIVE_RESULTS.md: what did not work, with numbers โ three heavy engineering efforts
abandoned on measurement, and the 9 instrument traps we fell into so you don't have to. ๐ชค
deepseek-ai/DeepSeek-V4-Flash-0731 (MIT) โ unsloth/DeepSeek-V4-Flash-0731-GGUF
(UD-IQ3_XXS, imatrix) โ this repo (attention families requantized; experts untouched).
sha256 538ec6932c50c6993f8a2cb18dc138e0fb5f6a7c2659e53a7566f1dc8ebb4344
size 103,159,533,696 bytes
Benchmarked on: AMD Ryzen AI Max+ 395 (gfx1151/RDNA 3.5), ROCm 7.1, 115.3 GiB unified, llama.cpp mainline HIP. Measured, documented, and calibrated โ enjoy. ๐