gemma-4-e2b-it-q8

Gemma 4 E2B it — INT8 Quantization (bitsandbytes)

Quantized version of google/gemma-4-e2b-it using bitsandbytes INT8. Tested on RTX 5090 (Blackwell, sm_120).

Benchmark Results

Tested across 4 categories (Math, Logic, Code, Science), 3 prompts each.
Greedy decoding (do_sample=False), 200 max new tokens.

Results by Category

Key Findings

• Quality: Q8 retains 92.9% token agreement with FP16 — nearly identical output
• VRAM: Saves 2.1 GB vs FP16 (1.3x compression)
• Speed: 14.5 tok/s — slower than Q4 due to bfloat16→float16 cast overhead in bitsandbytes on Blackwell GPUs
• Best for: Cases where quality is priority over speed

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model = AutoModelForCausalLM.from_pretrained(
    "MichaelLowrance/gemma-4-e2b-q8",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained("MichaelLowrance/gemma-4-e2b-q8")

messages = [{"role": "user", "content": "Hello!"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=200, do_sample=False)

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Hardware

Tested on: NVIDIA RTX 5090 (Blackwell, sm_120, 32GB GDDR7)
CUDA: 12.8 | Python: 3.12 | transformers: 5.6.0.dev