gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M

gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M

GGUF Q3_K_M weight-quantized variant of google/gemma-4-E2B-it optimised for use with RotorQuant KV cache compression via a dedicated llama.cpp fork.

Important: RotorQuant KV cache types (planar3, iso3) are not available in upstream llama.cpp, standard Ollama, or LM Studio. They require a specific llama.cpp fork. The GGUF file itself is a standard GGUF and works with any llama.cpp-compatible runtime using normal KV cache types (f16, q8_0, q4_0, etc.).

Overview

This model combines two independent compression techniques:

Quickstart

Option A — With RotorQuant KV cache (fork required)

You must build from the RotorQuant-enabled llama.cpp fork:

# Clone and build the fork
git clone https://github.com/johndpope/llama-cpp-turboquant.git
cd llama-cpp-turboquant && git checkout feature/planarquant-kv-cache

# CUDA (Windows/Linux)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Metal (Apple Silicon)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Run with RotorQuant KV cache
./build/bin/llama-cli -m gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

# Or run as a server
./build/bin/llama-server -m gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa --jinja

Option B — With standard llama.cpp / LM Studio / Ollama

The GGUF works as a normal quantised model. You won't get RotorQuant-specific KV cache benefits, but standard KV cache quantization (q8_0, q4_0) still reduces VRAM significantly.

llama.cpp (upstream)

llama-cli -m gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

LM Studio

1. Download the GGUF file and load in LM Studio.
2. Enable Developer Mode (Settings → Developer).
3. In the model loader's advanced settings, set Flash Attention to ON.
4. Set K Cache Quantization and V Cache Quantization to q8_0 (or q4_0 for more aggressive VRAM savings).
5. Note: LM Studio does not currently support RotorQuant's iso3 cache types. Track this feature request for updates.

Ollama

# Standard Ollama does not support RotorQuant cache types.
# Use with default or q8_0 KV cache via OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama run majentik/gemma-4-E2B-it-RotorQuant-GGUF-Q3_K_M

Specifications

What is RotorQuant?

RotorQuant is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors. It was developed as a faster, more parameter-efficient alternative to Google's TurboQuant (ICLR 2026).

Instead of applying a dense d×d random orthogonal rotation matrix (as TurboQuant does), RotorQuant uses lightweight block-diagonal rotations — independent 2D/4D rotations per pair/quartet — achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.

Benchmarks from the RotorQuant repository (Llama 3.1 8B, RTX 5090 — results will vary by model and hardware):

Note: These benchmarks are from the RotorQuant repository using Llama 3.1 8B on an RTX 5090. Performance on gemma-4-E2B-it will differ. Independent benchmarks for this specific model are welcome — please open a discussion if you have results to share.

Current Status of RotorQuant in the Ecosystem

Recommended Settings

For VRAM-constrained setups, standard q8_0 KV cache quantization already halves KV cache memory with negligible quality impact. Flash Attention should always be enabled — it is required for V cache quantization and improves memory efficiency regardless.

See Also

• RotorQuant GitHub
• llama-cpp-turboquant fork
• TurboQuant llama.cpp discussion
• TurboQuant paper (arXiv: 2504.19874)
• Base model: google/gemma-4-E2B-it
• gemma-4-E2B-it announcement