gemma-4-31B-it-W8A8-INT8

gemma-4-31B-it — AWQ 4-bit

Auto-quantized from google/gemma-4-31B-it by fastserve's self-quantization pipeline (publish/) — so it can be trusted the way a random community AWQ requant sometimes can't. Two community quants fastserve auto-detected during its own benchmark run looped garbage tokens instead of answering; every checkpoint published here passed an accuracy gate against its bf16 baseline first.

Quantization

• Method: AWQ (W8A8), group size 128, via llm-compressor
• Calibration: 256 samples from wikitext

Validation (accuracy gate — this is why you can trust it)

Within 0.1 (absolute) of the bf16 baseline, <30% degenerate (repeated-token loops). A checkpoint that failed this gate would not have been uploaded.

Serve it with fastserve

fastserve auto-detects this checkpoint — point it at the original model id and it finds this AWQ + wires up speculative decoding:

git clone https://github.com/jireh-father/fastserve && cd fastserve && ./install.sh

# serve an OpenAI-compatible API (auto-picks this AWQ checkpoint)
./fastserve serve google/gemma-4-31B-it

# or benchmark the speedup vs the naive baseline
./fastserve bench google/gemma-4-31B-it --compare-baseline

Then query it like any OpenAI endpoint:

curl localhost:8000/v1/completions \
  -d '{"model": "glenic/gemma-4-31B-it-W8A8-INT8", "prompt": "Q: What is 17*4?\nA:", "max_tokens": 64}'

Or serve directly with vLLM

pip install vllm
python -m vllm.entrypoints.openai.api_server --model glenic/gemma-4-31B-it-W8A8-INT8

License

Inherits the base model's license — see google/gemma-4-31B-it for terms.