Downloads · 30 days
12
3% of all-time downloads
Mistral-NeMo-Minitron-8B-Instruct-W4A16
useful-quants/Mistral-NeMo-Minitron-8B-Instruct-W4A16
Mistral-NeMo-Minitron-8B-Instruct-W4A16 is a text generation model from useful-quants. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Ampere-friendly serving build of nvidia/Mistral-NeMo-Minitron-8B-Instruct. Text-side linears are compressed-tensors W4A16.
All-time downloads
436
Public
Parameters
8.5B
6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors5.9 GB · 100%
Parameter types
How the weights are stored.
I327.3B · 87%
Try a prompt
Base models
Model card
From the Hugging Face model README
Mistral-NeMo-Minitron-8B-Instruct W4A16
Ampere-friendly serving build of nvidia/Mistral-NeMo-Minitron-8B-Instruct.
Text-side linears are compressed-tensors W4A16.
Stock proof
docker run --rm -it \
--gpus all \
--ipc=host \
-p 8001:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
nvidia/Mistral-NeMo-Minitron-8B-Instruct \
--served-model-name Mistral-NeMo-Minitron-8B-Instruct-stock \
--dtype bfloat16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.76
Use
docker run --rm -it \
--gpus all \
--ipc=host \
-p 8001:8000 \
-v /path/to/Mistral-NeMo-Minitron-8B-Instruct-W4A16:/model \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model /model \
--served-model-name Mistral-NeMo-Minitron-8B-Instruct-W4A16 \
--dtype bfloat16 \
--quantization compressed-tensors \
--max-model-len 4096
Smoke test
python verify.py --url http://localhost:8001/v1/completions
Notes
- Best fit: RTX 30xx/40xx Ampere cards.
- The repo includes the calibration corpus used for verification.
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- other
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026