Downloads · 30 days
35
6% of all-time downloads
DeepSeek-R1-Distill-Llama-8B-NVFP4
amer8/DeepSeek-R1-Distill-Llama-8B-NVFP4
DeepSeek-R1-Distill-Llama-8B-NVFP4 is a text generation model from amer8. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This is an NVFP4 quantized version of deepseek-ai/DeepSeek-R1-Distill-Llama-8B, optimized for NVIDIA GPUs using TensorRT-LLM.
All-time downloads
611
Public
Parameters
4.5B
6 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors6 GB · 100%
Parameter types
How the weights are stored.
U83.5B · 70%
Try a prompt
Base models
Model card
From the Hugging Face model README
DeepSeek-R1-Distill-Llama-8B-NVFP4
This is an NVFP4 quantized version of deepseek-ai/DeepSeek-R1-Distill-Llama-8B, optimized for NVIDIA GPUs using TensorRT-LLM.
Quantization Details
| Property | Value |
|---|---|
| Base Model | deepseek-ai/DeepSeek-R1-Distill-Llama-8B |
| Quantization Method | NVFP4 (2-bit weights + 4-bit scales) |
| Calibration Dataset | CNN/DailyMail |
| Calibration Samples | 512 |
| Tool | NVIDIA TensorRT Model Optimizer v0.35.0 |
| Export Format | Hugging Face |
Hardware Requirements
- GPU: NVIDIA GPU with FP4 support (Blackwell, Ada Lovelace, or newer)
- VRAM: ~40GB recommended
- Tested on: NVIDIA DGX Spark (GB10)
Usage
With TensorRT-LLM
from tensorrt_llm import LLM
llm = LLM(model="amer8/DeepSeek-R1-Distill-Llama-8B-NVFP4")
output = llm.generate("Paris is great because")
print(output)
With TensorRT-LLM Server
trtllm-serve amer8/DeepSeek-R1-Distill-Llama-8B-NVFP4 \
--backend pytorch \
--port 8000
Limitations
- Requires TensorRT-LLM for inference
- Not compatible with standard transformers library
- Optimized for NVIDIA GPUs only
License
This model inherits the license from the base model. See DeepSeek license.
Acknowledgments
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- mit
- Created
- Jan 24, 2026
- Updated
- Apr 11, 2026