DeepSeek-V4-Flash-Vision-Exp-GGUF

DeepSeek V4 Flash Vision Exp GGUF

WARNING: This is not currently supported by llama.cpp master and requires my experimental branch to use https://github.com/tarruda/llama.cpp/tree/dsv4-vision-exp-support

GGUF quantizations for deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.

DeepSeek published the original model weights in MXFP4, so the MXFP4 GGUFs in this repo are direct conversions of those original safetensors.

Quant Recipes

For every quant recipe, ffn_down_exps, ffn_gate_exps, and ffn_up_exps use the recipe's expert type. Compressor, indexer, token embedding, output, hyper-connection, router-bias, and token-to-expert mapping tensors retain the BF16, F32, or I32 types specified in scripts/quantize.sh. All three quants use the included 726-entry, 812-chunk importance matrix.

Usage

model="./IQ3_XXS/DeepSeek-V4-Flash-Vision-Exp-IQ3_XXS-00001-of-00004.gguf"
mmproj="./mmproj-DeepSeek-V4-Flash-Vision-Exp-BF16.gguf"

ctx=262144
parallel=1

llama-server --no-mmap --no-warmup \
  --model "$model" --mmproj "$mmproj" \
  ---ctx-size $ctx_size -np $parallel \
  --repeat-penalty 1.0  --presence-penalty 0.0 \
  --chat-template-kwargs '{"reasoning_effort": "max"}' \
  --reasoning-preserve \
  -b 2048 -ub 2048 \
  --cache-ram 1024 -ctxcp 12 \
  --temp 1.0 --top-p 1.0