Downloads · 30 days
6
23% of all-time downloads
ostquant_qwen3_4b_w4a4kv4_int4_v2
TrojAI/ostquant_qwen3_4b_w4a4kv4_int4_v2
ostquant_qwen3_4b_w4a4kv4_int4_v2 is a machine learning model from TrojAI. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Bit-packed to INT4 from the raw OSTQuant W4A4KV4 checkpoint. Cannot be loaded via vanilla AutoModelForCausalLM.frompretrained(...) — the OSTQuant runtime is required to unpack and evaluate correctly.
All-time downloads
26
Public
Parameters
961M
2.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.9 GB · 99%
Parameter types
How the weights are stored.
I32503M · 52%
Base models
Model card
From the Hugging Face model README
OSTQuant Qwen3-4B W4A4KV4 (INT4-packed, compact)
Bit-packed to INT4 from the raw OSTQuant W4A4KV4 checkpoint. Cannot be loaded
via vanilla AutoModelForCausalLM.from_pretrained(...) — the OSTQuant runtime
is required to unpack and evaluate correctly.
- Compressed from ~8.9 GB (raw .bin) to ~2.9 GB (3× compression).
- Weights on 4-bit grid. Activations AND KV cache also quantized to 4 bits at inference.
- Reference PPL when loaded via OSTQuant runtime: 17.02 (WikiText-2 seq 2048, raw .bin).
Source
- Raw: TrojAI/ostquant_qwen3_4b_w4a4kv4
- Packing script: rebuild_int4_from_raw.py
- Type
- qwen3
- Created
- Aug 11, 2026
- Updated
- Aug 11, 2026