DeepSeek-V4-Flash-0731-Ornstein-GGUF

DeepSeek-V4-Flash-0731-Ornstein-GGUF

Release status: all four GGUF tiers published and verified. Q8_0, Q6_K, Q4_K, and Q2_K are live with Hub SHA-256 and exact byte size independently matched. The companion BF16 safetensors checkpoint is also complete (48 shards; metadata commit 105ef75053e89b3700f8b6dc793fa7a8c04e12ee).

GGUF quantizations of GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein: a 43-layer Mixture-of-Experts (MoE) model with 256 routed experts (6 active per token) plus 1 shared expert, a 128-token sliding window, and a 1,048,576 token context window configured with YaRN RoPE scaling.

The LoRA adapter was trained on Fireworks AI and merged into the base model weights. These GGUF files contain 1,328 main-model tensors, including 129 routed tensors packing 256 experts each. Other two-dimensional floating-point matrices use Q8_0; one-dimensional values and integer controls retain their appropriate types. Multi-Token Prediction (MTP) / DSpark weights are omitted from these files.

The BF16 safetensors repository includes full Multi-Token Prediction (MTP) weights and is now complete. That release dequantizes remaining low-precision weights; it does not recover upstream FP4/FP8 precision.

Status

This checkpoint injects Ornstein thinking into DeepSeek-V4-Flash-0731. It is an early merge, not a finished quality release. Planned quality work uses RL environments and energy-based fine-tuning.

Evaluation & Verification Limits

Downstream Quality Evaluation

No downstream task benchmark or evaluation suite has been executed on these quantized checkpoints. Weight and file-format verification do not establish model quality or inference correctness.

Quantization Verification & Status

• Structural Integrity: The generated Q6_K GGUF has been verified for correct header magic (GGUF), tensor alignment, metadata schema, and tensor count (1,328 total tensors, 129 routed expert blocks packing 256 experts each, 3 I32 tensors). Note that structural GGUF proof confirms tensor serialization integrity, not complete end-to-end inference correctness.
• Conversion ladder: All four tiers (Q8_0, Q6_K, Q4_K, Q2_K) are published, with each Hub SHA-256 and exact byte size independently matched to its verified local artifact.
• Q8_0 SHA-256: 93a9890a2976651823c422ce3f9d0495812102ecfc6ce7dfd6a1fb5015f75314 (302,119,004,320 bytes).
• Q6_K SHA-256: 5ed95382fa3d3b9b94f1e22ee50ea184bde0b48ae9b4f9352df3d878d8136c5e (235,026,917,536 bytes).
• Q4_K SHA-256: 39814d4e30b723f88306449d1322895e1ce4d95dfe21812e62fda99332198cf3 (163,606,309,024 bytes).
• Q2_K SHA-256: 33c201aac413b307922abe1a10fd582c143b5787eee9d203e629ed88395dac3e (98,678,483,104 bytes).

Support this work

I'm a PhD student in visual neuroscience at the University of Toronto. Training and release compute is self-funded (rented H100s and a local DGX Spark). If these artifacts are useful, Ko-fi helps keep the experiments running.

Model details

Quant index

Choose a quantization tier that fits your available unified memory or VRAM with adequate headroom for KV cache and context execution. For systems such as the DGX Spark, Q2_K provides the highest execution headroom.

(Note: This is a custom Q4_K mixed recipe, not standard llama.cpp Q4_K_M. The …-Q4_K_M.gguf filename is a zero-storage Hub alias so the quant picker recognizes the tier; either filename downloads the same 163.6 GB artifact.)

Safe Download Instructions

After the corresponding file is published, download only the quantization you intend to use. Downloading the whole repository would fetch all four large files; Q8_0 alone is about 302 GB.

# Q2_K is the smallest tier; leave additional memory for runtime and context.
hf download GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein-GGUF DeepSeek-V4-Flash-0731-Ornstein-Q2_K.gguf --local-dir .

# Or download the Q4_K file
hf download GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein-GGUF DeepSeek-V4-Flash-0731-Ornstein-Q4_K.gguf --local-dir .

# Or download the Q6_K file
hf download GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein-GGUF DeepSeek-V4-Flash-0731-Ornstein-Q6_K.gguf --local-dir .

# Or download the Q8_0 file
hf download GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein-GGUF DeepSeek-V4-Flash-0731-Ornstein-Q8_0.gguf --local-dir .

Runtime Requirements & Usage

Runtime Requirement

The dsv4-mtp fork of am17an/llama.cpp is a source-supported runtime candidate: it has a real deepseek4 model loader and graph, CPU/CUDA support for Q6_K, Q4_K and Q2_K routed matrices, and Q8_0 auxiliary operations. These particular artifacts have not yet passed a full inference smoke test in that fork. Source support is not proof of successful model execution.

The inspected native antirez/ds4 runtime is not interchangeable with that fork: its fixed tensor-type requirements reject this Q8_0 auxiliary/embedding recipe, and its routed-format support differs. Do not assume stock llama.cpp, another fork, or a generic llama-cli command supports this release. Download examples are provided above; no unverified inference recipe is advertised.

Prompt Formatting

The upstream tokenizer does not provide a Jinja chat template. Do not assume a generic conversation mode will apply the correct formatting. Use the upstream Python encoding conventions in encoding/encoding_dsv4.py; see the companion HF card for an exercised encoding example.

Related Links

• BF16 safetensors: GestaltLabs/DeepSeek-V4-Flash-0731-Ornstein (dequantized; does not recover upstream FP4/FP8 precision)
• Upstream Base Model: deepseek-ai/DeepSeek-V4-Flash-0731
• Training Infrastructure: Fireworks AI

License

MIT License, inherited from the upstream DeepSeek-V4-Flash-0731 release.