Downloads · 30 days
20
4% of all-time downloads
caiovicentino1/Gemma-4-31B-it-HLWQ-Q5-Vision
Gemma-4-31B-it-HLWQ-Q5-Vision is a image-text-to-text model from caiovicentino1. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
[!IMPORTANT] Naming notice (2026-04-10). The "HLWQ" technique used in this model is being rebranded to HLWQ (Hadamard-Lloyd Weight Quantization). The change is only the name; the algorithm and the weights in this repo…
Downloads · 30 days
20
4% of all-time downloads
All-time downloads
522
Public
Repo size
21.9 GB
Likes
7
Public
Click a slice to open those files.
.pt21.8 GB · 100%
From the Hugging Face model README
[!IMPORTANT] Naming notice (2026-04-10). The "HLWQ" technique used in this model is being rebranded to HLWQ (Hadamard-Lloyd Weight Quantization). The change is only the name; the algorithm and the weights in this repository are unchanged.
The rebrand resolves a name collision with an unrelated, earlier KV cache quantization method also named HLWQ (Han et al., arXiv:2502.02617, 2025). HLWQ addresses weight quantization with a deterministic Walsh-Hadamard rotation and Lloyd-Max scalar codebook; Han et al.'s HLWQ addresses KV cache quantization with a random polar rotation. The two methods are technically distinct.
Existing loaders that load this repository by ID continue to work without changes. Future model uploads will use the HLWQ name.
Reference paper for this technique: arXiv:2603.29078 (v2 in preparation; v1 still uses the old name).
Multimodal (Image+Text → Text) on consumer GPUs with HLWQ.
| Component | Method | Result |
|---|---|---|
| Text weights | HLWQ Q5 + torchao INT4 | ~20 GB |
| Vision encoder | BF16 (full quality) | ~1.1 GB |
| KV Cache | HLWQ Q3 (5.3x compression) | longer context |
| Metric | Value |
|---|---|
| VRAM | 21.9 GB |
| Speed | 24.9 tok/s |
| Text layers | 412 INT4 |
| Vision layers | 190 BF16 |
| Compression | BF16 62.5 GB → 21.9 GB (2.9x) |
| Text test | ✅ "2+2 = 4" |
| Vision test | ✅ "Golden Gate Bridge" |

| GPU | VRAM | Fits? |
|---|---|---|
| RTX 4090 | 24 GB | ✅ Yes |
| RTX 5090 | 32 GB | ✅ Comfortable |
| L4 | 24 GB | ✅ Yes |
| A100 40GB | 40 GB | ✅ Plenty |
| T4 | 16 GB | ❌ Too small |
from transformers import AutoModelForMultimodalLM, AutoProcessor
import torch
MODEL = "google/gemma-4-31B-it"
processor = AutoProcessor.from_pretrained(MODEL)
# Load with streaming loader (see notebook for full code)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL, dtype=torch.bfloat16, device_map="cpu",
attn_implementation="sdpa",
)
# Apply PQ5 dequant + INT4 per-module (text only, vision stays BF16)
# ... see notebook ...
# Image + Text inference
messages = [{
"role": "user",
"content": [
{"type": "image", "url": "https://example.com/image.jpg"},
{"type": "text", "text": "Describe this image."},
]
}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_dict=True,
return_tensors="pt", add_generation_prompt=True).to("cuda")
output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
| Method | Bits | Compression | tok/s |
|---|---|---|---|
| FP16 (baseline) | 16 | 1.0x | 24.9 |
| HLWQ Q4 | 4 | 4.0x | 24.9 |
| HLWQ Q3 | 3 | 5.3x | 24.9 |
| HLWQ Q2 | 2 | 8.0x | 24.9 |
nn.Sequential wrapper — peak VRAM = final model only@article{polarquant2025,
title={HLWQ: Hadamard-Rotated Lloyd-Max Quantization for LLM Compression},
author={Vicentino, Caio},
journal={arXiv preprint arXiv:2603.29078},
year={2025},
url={https://arxiv.org/abs/2603.29078}
}
Built on Google's Gemma 4 (Apache 2.0). Quantization by HLWQ with torchao.
pip install git+https://github.com/caiovicentino/polarengine-vllm.git
from polarengine_vllm import HLWQModel
model = HLWQModel.from_pretrained("caiovicentino1/Gemma-4-31B-it-HLWQ-Q5-Vision")
print(model.generate("Hello, how are you?", max_new_tokens=100))
model = HLWQModel.from_pretrained("caiovicentino1/Gemma-4-31B-it-HLWQ-Q5-Vision", kv_cache_nbits=3)
# KV cache now uses 5.3x less memory — fit longer conversations!
print(model.generate("Explain quantum computing in detail.", max_new_tokens=500))
polarquant bench caiovicentino1/Gemma-4-31B-it-HLWQ-Q5-Vision --ppl --chart
polarquant demo caiovicentino1/Gemma-4-31B-it-HLWQ-Q5-Vision --share
Hadamard Rotation + Lloyd-Max Optimal Centroids
Unlike GGUF (uniform quantization), HLWQ places quantization levels where weight density is highest — mathematically proven optimal for Gaussian-distributed neural network weights.
HLWQ Q5 (cos_sim > 0.996) > GGUF Q5_K_M (~0.99) at same size