Downloads · 30 days
18
9% of all-time downloads
Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16
gemma-4-31B-it-scotoma-2-GPTQ-W8A16 is a text generation model from Captain1Q. Use it when you need the model to write or continue text. It is set up for transformers.
This is an 8-bit weight-only (W8A16) GPTQ quantization of ReadyArt/gemma-4-31B-it-scotoma-2, produced with llmcompressor and intended for fast, memory-efficient inference with vLLM.
Downloads · 30 days
18
9% of all-time downloads
All-time downloads
211
Public
Parameters
31.3B
33.7 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors33.7 GB · 100%
How the weights are stored.
I3229.3B · 94%
From the Hugging Face model README
This is an 8-bit weight-only (W8A16) GPTQ quantization of ReadyArt/gemma-4-31B-it-scotoma-2, produced with llmcompressor and intended for fast, memory-efficient inference with vLLM.
scotoma-2 is a 31B-parameter derivative of Google's gemma-4-31B-it. Per its model card, it applies a bounded refusal-direction edit (an abliteration LoRA projected through Gemma's J-Space) combined with several rounds of DPO preference training aimed at reducing repetitive stylistic tics (reflexive negation, em-dash asides, stacked adjectives) rather than removing the base model's safety behavior — the authors describe it explicitly as "not uncensored." See the base model's own card for full details on the method and its limitations; this repo only covers the quantization and adds nothing to the underlying behavior.
| Method | GPTQ (GPTQModifier, one-shot) via llmcompressor |
| Scheme | W8A16 — 8-bit integer weights, 16-bit activations |
| Format | compressed-tensors (native vLLM support) |
| Calibration data | 128 samples from HuggingFaceH4/ultrachat_200k (train_sft split) |
| Calibration sequence length | 2048 tokens |
| Layers excluded from quantization | lm_head, embedding layers, vision tower layers |
| Quantized on | NVIDIA A100 80GB |
Weight-only quantization keeps activations at full precision, which preserves accuracy well while roughly halving VRAM footprint relative to bf16. It primarily helps memory usage and single/low-batch latency; for very high-throughput serving, full activation quantization (W8A8) can offer more gains, but that path isn't well supported on Ampere-class GPUs, making W8A16 the practical choice here.
vllm serve Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16
vLLM will auto-detect the compressed-tensors quantization config from the checkpoint. On Ampere (A100/A10) GPUs this runs through the Marlin weight-only kernel automatically.
from vllm import LLM, SamplingParams
llm = LLM(model="Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
sampling_params = SamplingParams(temperature=1.0, max_tokens=512)
output = llm.generate(["Your prompt here"], sampling_params)
print(output[0].outputs[0].text)
The checkpoint also loads directly via transformers + compressed-tensors for testing outside vLLM, though vLLM is recommended for production serving speed.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
Quantized weights occupy roughly half the VRAM of the bf16 original (~31B params → ~31 GB in int8 vs. ~62 GB in bf16, before KV cache and activation overhead). Tested on a single A100 80GB; should also fit on a single A100 40GB or comparable Ampere GPU depending on context length and batch size.
google/gemma-4-31B-it and passed through ReadyArt/gemma-4-31B-it-scotoma-2). Review the base model's license and usage restrictions before deploying.llmcompressor GPTQModifier (W8A16)