Downloads · 30 days
14
2% of all-time downloads
donghyunli/Llama-2-7b-KronQ-W3A16-fake
Llama-2-7b-KronQ-W3A16-fake is a text generation model from donghyunli. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
Downloads · 30 days
14
2% of all-time downloads
All-time downloads
900
Public
Parameters
6.7B
27 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors13.5 GB · 100%
From the Hugging Face model README
Paper: arXiv:2607.07964 · Code: GitHub
⚠️ Fake-quant fp16 checkpoint. The 3-bit weights are stored in fp16 (KronQ does not pack int3), so this repo is the same size as bf16 — no compression or speedup, for PPL / accuracy reproduction only. For deployable low-bit, see the W4A16 / W2A16 (packed int4/int2) repos.
Llama-2-7b quantized to 3-bit weights with KronQ (Kronecker-factored Hessian quantization), exported as a standard fp16 model.
Perplexity: 5.83
Zero-shot accuracy:
| PIQA | ARC-E | ARC-C | HellaSwag | WinoGrande | BoolQ | OBQA | Average |
|---|---|---|---|---|---|---|---|
| 76.93 | 70.92 | 43.26 | 72.47 | 67.17 | 72.51 | 41.40 | 63.52 |
(lm-evaluation-harness, 0-shot. acc_norm for PIQA/HellaSwag/ARC/OBQA, acc for WinoGrande/BoolQ.)
Loads as a standard fp16 model — no KronQ code needed:
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("donghyunli/Llama-2-7b-KronQ-W3A16-fake", torch_dtype="float16").cuda()
tok = AutoTokenizer.from_pretrained("donghyunli/Llama-2-7b-KronQ-W3A16-fake")
Per-channel asymmetric W3, weight-only (a_bits=16), --alpha 0.25, bidirectional incoherence processing (BiIP), act_order, raw H_G. Calibrated on 128 WikiText-2 sequences.
Derivative of Llama-2-7b — subject to the Llama 2 Community License.