Qwen3.5-0.8B-AWQ-INT4

Qwen3.5-0.8B-AWQ-INT4

4-bit quantized version of Qwen/Qwen3.5-0.8B using AWQ (Activation-aware Weight Quantization) via llm-compressor, saved in the compressed-tensors format.

[!Note] This is a text-only quantized model. Vision capabilities from the original model have been removed.

Quantization Parameters

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "BlivionIaG/Qwen3.5-0.8B-AWQ-INT4"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM

from vllm import LLM

llm = LLM(
    model="BlivionIaG/Qwen3.5-0.8B-AWQ-INT4",
    quantization="compressed-tensors"
)

Requirements

pip install transformers compressed-tensors

For vLLM:

pip install vllm

Recommended Generation Settings

Hardware Requirements

About Qwen3.5

Qwen3.5 features a hybrid architecture combining Gated Delta Networks and Gated Attention, with 0.8B parameters and 262K context length. For full details, see the original model card.

Citation

@article{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2025}
}

License

This model is quantized from Qwen/Qwen3.5-0.8B which is licensed under the Apache License 2.0.

Copyright 2026 Alibaba Cloud.