Qwen3-0.6B-MYPTQ-w4g128

Qwen3-0.6B-MYPTQ-w4g128

4-bit weight-only quantization of Qwen/Qwen3-0.6B: group size 128 along in_features with integer zero-points, plus an INT4-packed ONNX graph (com.microsoft.MatMulNBits). The 28 decoder layers shrink 3.85x (840 -> 218 MB, 4.16 bit/weight); the tied embedding is left at 16-bit.

Built with MYPTQ, which holds the pipeline, the ablations and the full report.

Usage

model.safetensors stores the codes and zero-points packed two per byte, so plain AutoModelForCausalLM.from_pretrained cannot read it; load_realquant_model rebuilds the RealQuantLinear modules and unpacks them.

import torch
from huggingface_hub import snapshot_download
from quantlinear import load_realquant_model   # https://github.com/junuke/MYPTQ

local = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128")
model, tokenizer = load_realquant_model(local, dtype=torch.float16, device="cuda")

ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()
print(tokenizer.decode(model(input_ids=ids).logits[0, -1].argmax()))   # ' Paris'

The ONNX graph keeps its weights INT4-packed and needs ONNX Runtime >= 1.16. Its tensors live in onnx/model.onnx.data, which must sit next to onnx/model.onnx. Download the folder with local_dir=: the default cache stores files as symlinks into its blob store, and onnxruntime rejects external data whose real path leaves the model directory.

import numpy as np, onnxruntime as ort

graph = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128", allow_patterns=["onnx/*"], local_dir="onnx_model")
sess = ort.InferenceSession(f"{graph}/onnx/model.onnx",
                            providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
logits = sess.run(["logits"], {"input_ids": ids.cpu().numpy().astype(np.int64)})[0]

Activations are fp16. The graph takes input_ids [batch, seq], returns logits [batch, seq, vocab], and builds its causal mask internally (no KV cache input).

Measured results

1. Accuracy and general quality (fp16; MMLU: lm_eval mmlu, limit=0.1, max_length=512, batch_size=16; C4: 60 x 512 tokens, out of domain)

2. ONNX conversion quality (com.microsoft.MatMulNBits, INT4 packed; reference = PyTorch fp32 QuantLinear model)

3. Model size

Summary