Downloads · 30 days
16
9% of all-time downloads
Kiffaz11/ministral3-3b-reasoning-torchao-int4
ministral3-3b-reasoning-torchao-int4 is a image-text-to-text model from Kiffaz11. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This repository contains a TorchAO INT4 weight-only quantized checkpoint of:
Downloads · 30 days
16
9% of all-time downloads
All-time downloads
182
Public
Parameters
945M
2.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.9 GB · 99%
How the weights are stored.
I32481M · 51%
From the Hugging Face model README
This repository contains a TorchAO INT4 weight-only quantized checkpoint of:
mistralai/Ministral-3-3B-Reasoning-2512
Int4TilePackedTo4dTensorTest problem:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Expected answer: 72
Results:
| Model | Parsed answer | Correct |
|---|---|---|
| Original BF16 | 72 | yes |
| TorchAO INT4 | 72 | yes |
| Metric | Original BF16 | TorchAO INT4 |
|---|---|---|
| Checkpoint size GiB | 14.3391 | 2.6557 |
| Load time seconds | 6.9686 | 9.8574 |
| Generation latency seconds | 1.0092 | 1.0404 |
| Generated tok/s | 64.4074 | 53.8238 |
| Total tok/s including prompt | 165.4775 | 151.8600 |
| Peak VRAM allocated GiB | 15.9948 | 11.4769 |
| Peak VRAM reserved GiB | 16.0820 | 14.9023 |
| CPU RSS after load GiB | 8.0434 | 8.0436 |
This checkpoint is intended as a TorchAO quantized research artifact. The first validation target was successful reload, reduced checkpoint size, lower VRAM allocation, and correctness on a smoke arithmetic task.
The quantized model may not be faster than BF16 on every GPU/backend. For this checkpoint, the main gain is storage and allocated VRAM reduction.