Downloads · 30 days
16
10% of all-time downloads
huawei-csl/Qwen3-235B-A22B-3bit-SINQ
Qwen3-235B-A22B-3bit-SINQ is a text generation model from huawei-csl. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
<p align="center" <img src="logo.png" alt="Logo" style="max-width: 80%; height: auto;" </p
Downloads · 30 days
16
10% of all-time downloads
All-time downloads
162
Public
Repo size
104 GB
Likes
2
Public
Click a slice to open those files.
.safetensors104 GB · 100%
From the Hugging Face model README
This repository contains the official 3-bit quantized version of the Qwen3-235B-A22B model using the SINQ (Sinkhorn-Normalized Quantization) method.
SINQ is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.
To support the project please put a star ⭐ in the official SINQ github repository.
Qwen3-235B-A22B-3bit-SINQ Qwen/Qwen3-235B-A22BsinqBefore running the quantization script, make sure the SINQ library is installed. Installation instructions and setup details are available in the SINQ official github repository.
You can load and use the model with our wrapper based on the 🤗 Transformers library:
import torch
from transformers import AutoTokenizer
from sinq.patch_model import AutoSINQHFModel
model_name = "huawei-csl/Qwen3-235B-A22B-3bit-SINQ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
sinq_model = AutoSINQHFModel.from_quantized_safetensors(
model_name,
device="cuda:0",
compute_dtype=torch.bfloat16
)
prompt = "Explain neural network quantization in one sentence."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
with torch.inference_mode():
out_ids = sinq_model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
<details>
<summary><span style="font-size:1.1em; font-weight:bold;">🧩 Quantization Process</span></summary>
The quantized model was obtained using the SINQ quantization library, following the steps below:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from sinq.patch_model import AutoSINQHFModel
from sinq.sinqlinear import BaseQuantizeConfig
# Load base model
base_model_name = "Qwen/Qwen3-235B-A22B"
model = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype="float16")
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
# Apply 3-bit SINQ quantization
quant_cfg = BaseQuantizeConfig(
nbits=3, # quantization bit-width
group_size=64, # group size
tiling_mode="1D", # tiling strategy
method="sinq" # quantization method ("asinq" for the calibrated version)
)
qmodel = AutoSINQHFModel.quantize_model(
model,
tokenizer=tokenizer,
quant_config=quant_cfg,
compute_dtype=torch.bfloat16,
device="cuda:0"
)
</details> </br>Reproducibility Note: This model was quantized using the SINQ implementation from commit
14ad847of the SINQ repository.
If you find SINQ useful in your research or applications, please
@misc{muller2025sinq,
title={SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights},
author={Lorenz K. Muller and Philippe Bich and Jiawei Zhuang and Ahmet Celik and Luca Benfenati and Lukas Cavigelli},
year={2025},
eprint={2509.22944},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={http://arxiv.org/abs/2509.22944}
}