Llama-3.2-3B-Instruct-W4A16-G128

Llama-3.2-3B-Instruct-W4A16 (4-bit GPTQ, group size 128)

Built with Llama. A 4-bit weight-quantized version of meta-llama/Llama-3.2-3B-Instruct, produced so the model fits and runs on a 6 GB consumer GPU. The non-gated unsloth/Llama-3.2-3B-Instruct mirror was used as the quantization source.

TL;DR The original BF16 release is ~6.1 GB and does not comfortably fit 6 GB of VRAM with KV cache. Quantized to W4A16 (INT4, group size 128) with llmcompressor, it shrinks to ~2.3 GB (2.7x smaller) and serves in vLLM on an RTX 3050 6 GB Laptop GPU at ~66 tok/s single-stream and ~1,400 tok/s batched, with reasoning intact.

How it was made

• Method: GPTQ, W4A16 scheme (4-bit symmetric INT weights, FP16 activations), group size 128, lm_head left unquantized.
• Tool: llmcompressor 0.12, output in compressed-tensors (pack-quantized) format.
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k @ 2048 tokens.
• Hardware: RTX 3050 6 GB. Loaded on CPU and quantized layer-by-layer, peak VRAM ~2 GB.

Measured results (RTX 3050 6 GB, vLLM 0.22, W4A16)

Quality spot-check (greedy): correctly computes 3^100 mod 7 = 4 with full Fermat's-little-theorem working, and lists the first ten primes correctly. This is a spot check, not a full eval. 4-bit quantization is lossy in general; these specific prompts held up.

Part of a 3-model comparison

Quantized with the same pipeline as two other models and benchmarked together on one 6 GB card:

Same ~2.2 GB on disk, but servable context varies 11x across the three. Full writeup and code linked below.

Usage (vLLM)

vllm serve syedazeez/Llama-3.2-3B-Instruct-W4A16-G128 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.88

from vllm import LLM, SamplingParams

llm = LLM(model="syedazeez/Llama-3.2-3B-Instruct-W4A16-G128",
          max_model_len=4096, gpu_memory_utilization=0.88)
out = llm.chat(
    [[{"role": "user", "content": "State Fermat's little theorem and compute 3^100 mod 7."}]],
    SamplingParams(temperature=0, max_tokens=400),
)
print(out[0].outputs[0].text)

License & attribution

This is a derivative of Meta's Llama 3.2 and is distributed under the Llama 3.2 Community License. "Built with Llama." All credit for the model goes to Meta; only the 4-bit quantization was performed here. Use is subject to the Llama 3.2 license and its Acceptable Use Policy.