Downloads · 30 days
19
13% of all-time downloads
lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit
qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit is a image-text-to-text model from lew96123. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This repository is a standalone quantized release of Qwen/Qwen3.5-0.8B published by lew96123. The previous model card was only a short summary; this one is intentionally detailed so the repo page itself explains what…
Downloads · 30 days
19
13% of all-time downloads
All-time downloads
152
Public
Repo size
124 MB
Likes
0
Public
Click a slice to open those files.
.safetensors112 MB · 83%
From the Hugging Face model README
This repository is a standalone quantized release of Qwen/Qwen3.5-0.8B published by lew96123.
The previous model card was only a short summary; this one is intentionally detailed so the repo page itself explains what was stored, how it was produced, what was evaluated, and what trade-offs showed up.
Qwen/Qwen3.5-0.8Bimage-text-to-text / image-text-to-text| File | Purpose |
|---|---|
turboquant_weights.safetensors | Packed checkpoint payload containing quantized weights (and any passthrough tensors stored directly). |
quant_manifest.json | Full manifest with per-parameter metadata, storage prefixes, tensor shapes, and effective bit-width decisions. |
eval_summary.json | Serialized evaluation metrics for the base model and the quantized artifact. |
sample_generations.json | Deterministic sample generations generated from the stored prompts in the pipeline config. |
load_quantized.py | Self-contained Python loader that reconstructs the quantized model from this repository alone. |
requirements.txt | Minimal dependency hints for loading this artifact. |
config.json, tokenizer files, generation config, and related support files | Upstream support files copied so the repo can be loaded without separately downloading missing config assets. |
This release uses TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874, Zandieh et al. 2025) -- online vector quantization with near-optimal distortion rate. Each weight row is treated as a d-dimensional vector and quantized using a random rotation followed by optimal scalar quantization on the induced Beta distribution coordinates (Lloyd-Max codebook).
Allocator-driven mixed precision is disabled for this artifact, so the release stays uniform apart from any protected-weight rules.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874, Apr 2025; Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni).SpinQuant (arXiv:2405.16406) -- learned rotations for PTQ.APTQ (arXiv:2402.14866) and BAQ (arXiv:2506.05664) -- sensitivity-based mixed precision / bit allocation.GuidedQuant (arXiv:2505.07004), LeanQuant (arXiv:2407.10032), and D^2Quant (arXiv:2602.02546) -- stronger sub-4-bit weight-only PTQ strategies.PolarQuant (arXiv:2603.29078) and ParoQuant (arXiv:2511.10645) -- newer rotation / preprocessing directions for low-bit compression.Some tensors are intentionally treated as sensitive and do not follow the same policy as the main body weights.
Protected tensors quantized in this artifact
Protected tensors kept as passthrough
Mixed-precision tensors introduced by the protection rule
| Quantity | Value |
|---|---|
| Original bytes | 1,746,882,752 |
| Stored bytes | 111,412,032 |
| Bytes saved | 1,635,470,720 |
| Compression ratio | 15.679x |
| Quantized tensor count | 265 |
| Passthrough tensor count | 223 |
| Quantized scalar values | 873,250,816 |
| Passthrough scalar values | 187,968 |
The table below shows the largest tensors recorded in the manifest. This is helpful when auditing whether the storage budget is dominated by a few very large matrices or by a broad spread of medium-sized tensors.
| Tensor | Kind | Effective bit-width | Shape | Values |
|---|---|---|---|---|
model.language_model.embed_tokens.weight | quantized | 1 | 248320x1024 | 254,279,680 |
model.visual.merger.linear_fc1.weight | quantized | 1 | 3072x3072 | 9,437,184 |
model.language_model.layers.0.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.1.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.10.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.12.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.13.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.14.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.16.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.17.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.18.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
model.language_model.layers.2.linear_attn.in_proj_qkv.weight | quantized | 1 | 6144x1024 | 6,291,456 |
The dataset below is used in this repo as a sanity / regression test set only. It is useful for catching catastrophic degradation and comparing close variants under a fixed quick pass, but it is not enough to support broad quality claims.
| Item | Value |
|---|---|
| Dataset | wikitext:wikitext-2-raw-v1 |
| Dataset repo | wikitext |
| Dataset config | wikitext-2-raw-v1 |
| Dataset role in this repo | sanity-test |
| Split | test |
| Max context length | 512 |
| Sliding-window stride | 256 |
| Max eval windows | full pass |
| Logit comparison prompts | 4 |
| Sample generation prompts | 3 |
| Generation mode | deterministic greedy decoding |
| Max generated tokens | 64 |
| Metric | Base model | Quantized model | Notes |
|---|---|---|---|
| Average NLL | 2.906366 | 14.093450 | Lower is better |
| Perplexity | 18.2902 | 1320406.8612 | Lower is better |
| Tokens scored | 295,893 | 295,893 | Sliding-window effective tokens |
| Windows scored | 1,160 | 1,160 | Reported when dataset evaluation is used |
| Mean last-token logit cosine similarity | N/A | -0.060028 | Higher is better |
from load_quantized import load_quantized_model, load_tokenizer
model, manifest = load_quantized_model(".", device="cuda")
tokenizer = load_tokenizer(".")
prompt = "Explain why quantization can reduce storage requirements."
rendered = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False,
add_generation_prompt=True,
) if hasattr(tokenizer, "apply_chat_template") else prompt
batch = tokenizer(rendered, return_tensors="pt").to(model.device)
output = model.generate(**batch, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Inline sample generations are omitted for this release because the fixed prompt set produced severely degraded outputs under this quantization setting (perplexity ratio vs base: 72192.0x, mean last-token logit cosine: -0.060). The deterministic outputs are still preserved in sample_generations.json; when available, the unsanitized decode is retained in the raw_response field for auditability.
The pipeline fields below are the key knobs that shaped this artifact.
{
"base_model_id": "Qwen/Qwen3.5-0.8B",
"loader_kind": "image-text-to-text",
"quantization_method": "turboquant_mse",
"primary_bit_width": 1,
"effective_quantized_bit_widths": [
1
],
"seed": 42,
"grid_size": 8193,
"iterations": 96,
"eval_dataset": "wikitext:wikitext-2-raw-v1",
"eval_dataset_role": "sanity-test",
"eval_stride": 256,
"max_eval_length": 512,
"enable_sensitivity_allocation": false,
"allocator_name": "disabled",
"allocation_method": "uniform",
"allocation_allowed_bit_widths": [],
"calibration_texts": [],
"max_generation_tokens": 64,
"protected_weight_exact_names": [],
"protected_weight_name_fragments": []
}
Qwen/Qwen3.5-0.8Bapache-2.0lew96123lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bitquant_manifest.jsoneval_summary.jsonsample_generations.json