Phi-3.5-mini-instruct-W4A16-G128

Phi-3.5-mini-instruct-W4A16 (4-bit GPTQ, group size 128)

A 4-bit weight-quantized version of microsoft/Phi-3.5-mini-instruct, produced so the model fits and runs on a 6 GB consumer GPU.

TL;DR The original BF16 release is ~7.6 GB and does not fit 6 GB of VRAM. Quantized to W4A16 (INT4, group size 128) with llmcompressor, it shrinks to ~2.3 GB (3.35x smaller) and serves in vLLM on an RTX 3050 6 GB Laptop GPU at ~69 tok/s single-stream, with its reasoning intact.

How it was made

• Method: GPTQ, W4A16 scheme (4-bit symmetric INT weights, FP16 activations), group size 128, lm_head left unquantized.
• Tool: llmcompressor 0.12, output in compressed-tensors (pack-quantized) format.
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k @ 2048 tokens.
• Hardware: RTX 3050 6 GB. Loaded on CPU and quantized layer-by-layer (sequential GPU on-loading), peak VRAM ~1.6 GB.

Measured results (RTX 3050 6 GB, vLLM 0.22, W4A16)

Quality spot-check (greedy): correctly computes 3^100 mod 7 = 4 with full Fermat's-little-theorem working, and lists the first ten primes correctly. This is a spot check, not a full eval. 4-bit quantization is lossy in general; these specific prompts held up.

Part of a 3-model comparison

This model was quantized with the same pipeline as two others and benchmarked together on one 6 GB card:

Same ~2.2 GB on disk, but servable context ranges 11x across the three (Phi is the heaviest KV footprint of the set, so it has the smallest context budget on a fixed card). Full writeup and code linked below.

Usage (vLLM)

vllm serve syedazeez/Phi-3.5-mini-instruct-W4A16-G128 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.88

from vllm import LLM, SamplingParams

llm = LLM(model="syedazeez/Phi-3.5-mini-instruct-W4A16-G128",
          max_model_len=4096, gpu_memory_utilization=0.88)
out = llm.chat(
    [[{"role": "user", "content": "State Fermat's little theorem and compute 3^100 mod 7."}]],
    SamplingParams(temperature=0, max_tokens=400),
)
print(out[0].outputs[0].text)

On a 6 GB card, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True so CUDA-graph capture fits.

License & attribution

Released under the MIT License, the same as the original microsoft/Phi-3.5-mini-instruct. All credit for the model goes to Microsoft. Only the 4-bit quantization was performed here.