Downloads · 30 days
695
16% of all-time downloads
Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4
DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4 is a text generation model from Baekpica. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
Downloads · 30 days
695
16% of all-time downloads
All-time downloads
4.4K
Public
Parameters
61.3B
70.4 GB on disk
Likes
6
Public
Click a slice to open those files.
.safetensors70.1 GB · 100%
How the weights are stored.
U858.5B · 95%
From the Hugging Face model README
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.
<a href="https://www.buymeacoffee.com/baekpica" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;"></a> <a href="https://github.com/sponsors/Baekpica" target="_blank"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="Sponsor Baekpica on GitHub" style="height: 60px !important;width: 217px !important;"></a>
This is the predominantly weight-only NVFP4A16 version of
Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-BF16.
A16) on the NVFP4 paths; this is not
the W4A4 NVFP4 preset. The grouped attention WO_A compatibility path is
the explicit mixed-precision exception described below.llm-compressor model-free PTQ,
scheme="NVFP4A16", using data-free RTN. The shorter repository suffix is
NVFP4; the checkpoint is still weight-only W4A16, not W4A4.quant_config=None. The 43 grouped WO_A attention projections use dynamic
W8A8 E4M3 block FP8 (128×128 weight blocks, 128-element activation groups)
because the native CUDA inverse-RoPE output einsum directly consumes that
grouped FP8 layout. Linear gate_proj weights,
including shared-expert projections, remain part of the NVFP4 fused gate/up
quantization path.nvfp4-pack-quantized checkpoints.The parent model has 119,821,633,111 logical main-model parameters, 43 decoder layers, 104 routed experts per layer, top-6 routing, and approximately 13.802B active parameters/token. DSpark/MTP auxiliary weights are not included.
Build provenance is recorded in nvfp4a16_provenance.json; the checkpoint
header/config audit is in checkpoint_audit.json. See the BF16 parent model card
for the direct-saliency REAM and calibration details.
The single-GPU configuration below was verified on NVIDIA GB10 using the
Baekpica/vllm branch
deepseek-v4-v0.26.0,
based on the upstream v0.26.0 tag. The compatibility changes are pinned in
commit 622008c4e:
# Grouped FP8 WO_A uses the DeepGEMM scale attribute after loading.
- weight_scale
+ weight_scale_inv
# The fused CUDA router has no 104-expert instantiation.
- if current_platform.is_xpu():
+ if current_platform.is_xpu() or gating_output.shape[-1] == 104:
Install the fork with the precompiled vLLM extension, then install the matching FlashInfer packages. The CUDA-specific index is required for the JIT-cache wheel used by the validated CUDA 13.0 environment.
git clone --branch deepseek-v4-v0.26.0 --single-branch \\
https://github.com/Baekpica/vllm.git
cd vllm
uv venv --python 3.12
source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
uv pip install \\
flashinfer-python==0.6.14 \\
flashinfer-cubin==0.6.14 \\
'flashinfer-jit-cache==0.6.14+cu130' \\
--extra-index-url https://flashinfer.ai/whl \\
--extra-index-url https://flashinfer.ai/whl/cu130
Use an nvcc/header pair that matches the PyTorch CUDA build. The validated
DGX Spark environment used /usr/local/cuda-13.0:
export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"
vllm serve Baekpica/DeepSeek-V4-Flash-0731-120B-REAM-104E-NVFP4 \\
--port 8001 \\
--optimization-level 3 \\
--max-model-len 196608 \\
--max-num-seqs 2 \\
--max-num-batched-tokens 16384 \\
--gpu-memory-utilization 0.92 \\
--enable-prefix-caching \\
--enable-chunked-prefill \\
--kv-cache-dtype fp8 \\
--moe-backend auto \\
--attention-backend flashinfer \\
--tokenizer-mode deepseek_v4 \\
--reasoning-parser deepseek_v4 \\
--tool-call-parser deepseek_v4 \\
--enable-auto-tool-choice \\
--quantization compressed-tensors \\
--language-model-only
The checkpoint quantization_config declares the packed format and ignore
list. No runtime adapter or model-repository custom code is required.
The unquantized output tensor intentionally retains the original DeepSeek-V4
checkpoint name head.weight; vLLM's native DeepSeek-V4 mapper exposes it as
lm_head.weight at runtime. Tensor paths otherwise use the original native
DeepSeek-V4 layout expected by that mapper; no adapter or custom model file is
needed.
The compatibility branch and 196,608-token launch configuration above were verified to load and serve this checkpoint on DGX Spark / NVIDIA GB10.
Validated on 4× NVIDIA B200 with vLLM 0.22.0, tensor parallel 4, expert
parallel enabled, 8,192-token context, FP8 KV cache, and CUDA graphs enabled.
The server completed weight loading, the 104-expert routing warmup, CUDA graph
capture, /health (HTTP 200), model discovery, and four OpenAI-compatible chat
requests. Raw responses and timings are in vllm_serving_smoke_2048.json.
| Case | Result | Completion |
|---|---|---|
| Korean Python | Format rubric 4/4; semantic review found incorrect edge-case assertions | 490 tokens, stop |
| Reservoir math | Failed; arithmetic error followed by repetition | 2,048 tokens, length |
| 5,810-token needle | Exact retrieval | 55 tokens, stop |
| Strict JSON | Exact output | 29 tokens, stop |
This smoke test demonstrates that the standalone repository serves end to end; it is not evidence of benchmark-level quality. In particular, the Korean code was fluent and structurally complete but should not be treated as a correctness pass because some generated test expectations were inconsistent with the problem definition.
This is RTN weight-only quantization of an expert-merged model. Quantization and expert merging can both affect quality. The 2,048-token serving smoke exposed a basic arithmetic failure/repetition and incorrect coding edge-case assertions, while long-context retrieval and strict JSON succeeded. Korean fluency and surface code structure therefore do not guarantee logical correctness. Run task-specific evaluations and sandbox generated code before production use. The source model's MIT license and usage considerations apply.