Downloads · 30 days
17
17% of all-time downloads
DeepSeek-R1-Distill-Llama-70B-MXFP8
amanwalksdownthestreet/DeepSeek-R1-Distill-Llama-70B-MXFP8
DeepSeek-R1-Distill-Llama-70B-MXFP8 is a text generation model from amanwalksdownthestreet. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
MXFP8 quantization of deepseek-ai/DeepSeek-R1-Distill-Llama-70B. Weights only — prompt format, chat template, <think reasoning behavior, and license (MIT) are the base model's.
All-time downloads
99
Public
Parameters
72.7B
74.8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors74.8 GB · 100%
Parameter types
How the weights are stored.
F8_E4M368.5B · 94%
Try a prompt
Base models
Model card
From the Hugging Face model README
DeepSeek-R1-Distill-Llama-70B-MXFP8
MXFP8 quantization of deepseek-ai/DeepSeek-R1-Distill-Llama-70B. Weights only — prompt format, chat template, <think> reasoning behavior, and license (MIT) are the base model's.
Requires Blackwell (compute capability 12.0). MXFP8 needs Blackwell tensor-core support in vLLM; does not run on Ada/Hopper.
- Weights: MXFP8 (group size 32),
compressed-tensorsformat (llm-compressor 0.12.1) - Not quantized:
lm_head, embeddings. KV cache: BF16 - Size: 70 GB, 11 shards (base BF16 ≈ 141 GB)
- Calibration: 1024 samples,
neuralmagic/calibration(LLM), max seq 1024
Serving
vllm serve <path-or-repo> --quantization compressed-tensors \
--max-model-len 49152 --gpu-memory-utilization 0.95
Reasoning model: expect <think>…</think> traces per the base model's chat template (--reasoning-parser deepseek_r1 in vLLM separates them).
Evals (lm-evaluation-harness 0.4.11 on vLLM)
| Benchmark | DeepSeek paper (BF16) | MXFP8 (this repo) |
|---|---|---|
| MMLU-Pro (5-shot, non-thinking, /v1/completions) | not published | 67.50 ±0.41 |
| GPQA-Diamond | 65.2 | pending* |
| AIME24 | 70.0 (multi-sample avg) | pending* |
* The first GPQA/AIME24 run was invalidated by eval-infrastructure contention (195/198 GPQA responses returned empty — two sweeps fought over one endpoint); numbers will land after a clean re-run. The MMLU-Pro row is a deliberate non-thinking self-baseline (raw completions, no chat template, no <think> budget) — do not compare it against thinking-mode numbers.
All credit for the model to DeepSeek; this repo is a Blackwell-native repack.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- mit
- Languages
- en
- Created
- Jul 10, 2026
- Updated
- Jul 13, 2026