Downloads · 30 days
408
11% of all-time downloads
gemma-4-31B-it-AutoRound-GGUF
sphaela/gemma-4-31B-it-AutoRound-GGUF
gemma-4-31B-it-AutoRound-GGUF is a machine learning model from sphaela. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This repository contains GGUF quantized versions of google/gemma-4-31B-it created using Intel's AutoRound quantization method.
All-time downloads
3.9K
Public
Repo size
292 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.gguf292 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 17, 2026
- Updated
- Apr 24, 2026
- SHA
- bd30fe63
Base models
Model card
From the Hugging Face model README
Gemma-4-31B-it GGUF (AutoRound Quantized)
This repository contains GGUF quantized versions of google/gemma-4-31B-it created using Intel's AutoRound quantization method.
Quantization Details
The models were quantized using various schemes provided by the auto-round tool. For better compatibility and smaller size, we provide unified multimodal projector (mmproj) files in F16, BF16, and F32 formats.
Files and Sizes
| File Name | Quant Type | Size | Description |
|---|---|---|---|
gemma-4-31B-it-Q2_K_S.gguf | Q2_K_S | 11 GB | Extremely high compression, significant quality loss. |
gemma-4-31B-it-Q2_K_MIXED.gguf | Q2_K_MIXED | 17 GB | Recommended high-compression option. Uses Q4 for KV cache with good quality. |
gemma-4-31B-it-Q3_K_S.gguf | Q3_K_S | 13 GB | Very high compression, notable quality loss. |
gemma-4-31B-it-Q3_K_M.gguf | Q3_K_M | 13 GB | Balanced 3-bit quantization. |
gemma-4-31B-it-Q3_K_L.gguf | Q3_K_L | 13 GB | High quality 3-bit quantization. |
gemma-4-31B-it-Q4_0.gguf | Q4_0 | 17 GB | Standard 4-bit quantization, good balance. |
gemma-4-31B-it-Q4_1.gguf | Q4_1 | 19 GB | Higher quality 4-bit quantization than Q4_0. |
gemma-4-31B-it-Q4_K_S.gguf | Q4_K_S | 17 GB | Small 4-bit K-quant, good efficiency. |
gemma-4-31B-it-Q4_K_M.gguf | Q4_K_M | 17 GB | Recommended 4-bit K-quant, excellent balance. |
gemma-4-31B-it-Q5_0.gguf | Q5_0 | 20 GB | Standard 5-bit quantization, very high quality. |
gemma-4-31B-it-Q5_1.gguf | Q5_1 | 22 GB | Higher quality 5-bit quantization than Q5_0. |
gemma-4-31B-it-Q5_K_S.gguf | Q5_K_S | 20 GB | Small 5-bit K-quant, very high quality. |
gemma-4-31B-it-Q5_K_M.gguf | Q5_K_M | 20 GB | Recommended 5-bit K-quant, near-lossless. |
gemma-4-31B-it-Q6_K.gguf | Q6_K | 24 GB | 6-bit K-quant, virtually indistinguishable from F16. |
gemma-4-31B-it-Q8_0.gguf | Q8_0 | 31 GB | 8-bit quantization, near-lossless. |
mmproj-model-f16.gguf | F16 | 1.2 GB | Unified Projector in Float16 format. |
mmproj-model-bf16.gguf | BF16 | 1.2 GB | Unified Projector in BFloat16 format. |
mmproj-model-f32.gguf | F32 | 2.2 GB | Unified Projector in Float32 format. |
Generate the Model
The models were generated using Intel's AutoRound with the following command:
auto-round --model google/gemma-4-31B-it --output_dir ./quantized/ --scheme <SCHEME> --iters 0
Note: To reproduce this quantization, you need the following Pull Request:
- Intel AutoRound PR #1655
- And the specific fix mentioned in this comment.
Usage with llama.cpp
These models can be used with llama.cpp. For multimodal usage, you must specify the projector file:
./llama-cli -m gemma-4-31B-it-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image your_image.jpg -p "Describe this image."
About AutoRound
AutoRound is an advanced quantization technique from Intel that aims to minimize accuracy loss through automated rounding optimization.
- License
- apache-2.0
- Languages
- multilingual
- Created
- Apr 17, 2026
- Updated
- Apr 24, 2026