Qwen3.5-4B-autoround-W4A16

kaitchup/Qwen3.5-4B-autoround-W4A16

Qwen3.5-4B-autoround-W4A16 is a machine learning model from kaitchup. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

<div align="center" <img src="https://cdn-uploads.huggingface.co/production/uploads/64b93e6bd6c468ac7536607e/mj6xac74jHGLqymiovObc.png" alt="The Kaitchup -- AI on a Budget" style="width: 100%; max-width: 100%; height:…

Downloads · 30 days

1% of all-time downloads

All-time downloads

1.6K

Public

Parameters

1.7B

4.6 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors4.6 GB · 100%

Parameter types

How the weights are stored.

BF16971M · 59%

Base models

Qwen/Qwen3.5-4B

Model card

From the Hugging Face model README

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/64b93e6bd6c468ac7536607e/mj6xac74jHGLqymiovObc.png" alt="The Kaitchup -- AI on a Budget" style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;" /> <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;"> <a href="https://kaitchup.substack.com/subscribe"><strong>Subscribe and Support</strong></a> </div> </div>

This is Qwen/Qwen3.5-4B quantized with AutoRound to W4A16 (INT). The model is compatible with vLLM (tested: v0.16.1rc1). Tested with an H200. Currently under evaluation.

Developed by: The Kaitchup

Instructions

uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
uv pip install git+https://github.com/huggingface/transformers.git

vllm serve [this model ID]  --max-model-len 262144 --reasoning-parser qwen3

Acknowledgments

Thank you Verda for providing the needed compute. I used their H200s. Verda is a European, AI-focused cloud and GPU infrastructure provider with sovereignty, sustainability, data privacy, and performance at its core. Check them out if interested.

Type: qwen3_5
License: apache-2.0
Created: Mar 3, 2026
Updated: Mar 3, 2026