Downloads · 30 days
24
8% of all-time downloads
Llama-3.1-8B-Instruct-MR-GPTQ-mxfp
ISTA-DASLab/Llama-3.1-8B-Instruct-MR-GPTQ-mxfp
Llama-3.1-8B-Instruct-MR-GPTQ-mxfp is a image-text-to-text model from ISTA-DASLab. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as llama3.1.
This model was obtained by quantizing the weights of Llama-3.1-8B-Instruct to MXFP4 data type. This optimization reduces the number of bits per parameter from 16 to 4.25, reducing the disk size and GPU memory requirem…
All-time downloads
306
Public
Parameters
4.8B
5.8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors5.8 GB · 100%
Parameter types
How the weights are stored.
U83.7B · 78%
Try a prompt
Base models
Model card
From the Hugging Face model README
Llama-3.1-8B-Instruct-MR-GPTQ-mxfp
Model Overview
This model was obtained by quantizing the weights of Llama-3.1-8B-Instruct to MXFP4 data type. This optimization reduces the number of bits per parameter from 16 to 4.25, reducing the disk size and GPU memory requirements by approximately 73%.
Usage
MR-GPTQ quantized models with QuTLASS kernels are supported in the following integrations:
transformerswith these features:- Available in
main(Documentation). - RTN on-the-fly quantization.
- Pseudo-quantization QAT.
- Available in
vLLMwith these features:- Available in this PR.
- Compatible with real quantization models from
FP-Quantand thetransformersintegration.
Evaluation
This model was evaluated on a subset of OpenLLM v1 benchmarks and Platinum bench. Model outputs were generated with the vLLM engine.
OpenLLM v1 results
| Model | MMLU‑CoT | GSM8k | Hellaswag | Winogrande | Average | Recovery (%) |
|---|---|---|---|---|---|---|
meta‑llama/Llama 3.1‑8B‑Instruct | 0.7276 | 0.8506 | 0.8001 | 0.7790 | 0.7893 | – |
ISTA‑DASLab/Llama‑3.1‑8B‑Instruct‑MR‑GPTQ‑mxfp | 0.6754 | 0.7892 | 0.7737 | 0.7324 | 0.7427 | 94.09 |
Platinum bench results
Below we report recoveries on individual tasks as well as the average recovery.
Recovery by Task
| Task | Recovery (%) |
|---|---|
| SingleOp | 97.94 |
| SingleQ | 95.95 |
| MultiArith | 98.22 |
| SVAMP | 95.08 |
| GSM8K | 93.69 |
| MMLU-Math | 80.54 |
| BBH-LogicalDeduction-3Obj | 89.87 |
| BBH-ObjectCounting | 82.03 |
| BBH-Navigate | 90.66 |
| TabFact | 86.92 |
| HotpotQA | 96.81 |
| SQuAD | 98.46 |
| DROP | 94.33 |
| Winograd-WSC | 89.47 |
| Average | 92.14 |
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- llama
- License
- llama3.1
- Created
- Oct 2, 2025
- Updated
- Oct 3, 2025