Qwen3.6-27B-4bit-text-only

<div align="center"> <img width="300px" src="https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.6/logo.png"> <h1>Qwen3.6-27B-4bit-text-only</h1> <p>4-bit NF4 Quantized Text-Only Version (No Vision)</p> </div>

---

[!WARNING] TEXT-ONLY MODEL: This quantized version does NOT have vision/multimodal capabilities. The original Qwen3.6-27B is a multimodal model (text + image), but quantization with bitsandbytes + AutoModelForCausalLM strips vision components. For vision support, use the original model.

[!IMPORTANT] This is a 4-bit NF4 quantized version of Qwen3.6-27B. The 17GB disk size is compressed - actual GPU memory usage is ~16GB with Transformers or ~14-15GB with vLLM.

Recommended: 24GB+ GPU (RTX 4090, A5000) Minimum: 16GB GPU with vLLM (not Transformers)

Model Summary

[!NOTE] Why Text-Only? The original Qwen3.6-27B is a multimodal model (Qwen3_5ForConditionalGeneration) that supports image+text input. However, quantizing with bitsandbytes + AutoModelForCausalLM converts it to text-only (Qwen3_5ForCausalLM). The config has been manually fixed to Qwen3_5ForConditionalGeneration, but vision components are not included in the quantized weights.

Installation

pip install transformers bitsandbytes accelerate torch

Quick Start

Transformers API

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "/path/to/Qwen3.6-27B-4bit"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)

# Generate response
prompt = "Write a binary search function in Python"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

With Chat Template

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "/path/to/Qwen3.6-27B-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="cuda",
    torch_dtype=torch.bfloat16,
)

# Use chat template for conversation
messages = [{"role": "user", "content": "Implement a stack class in Python"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

vLLM (Recommended for Production)

# Install vLLM
pip install vllm>=0.5.0

# Serve the model
vllm serve Qwen3.6-27B-4bit --quantization nf4 --dtype half

# Or run with OpenAI-compatible API
vllm serve Qwen3.6-27B-4bit --quantization nf4 --host 0.0.0.0 --port 8000

GPU Requirements

Performance Testing Results

Verified on A100 80GB PCIe GPU.

Code Generation Tasks

Mathematical Reasoning

Agentic Coding Patterns

Accuracy Comparison

Compared against original FP16 model:

Token Similarity: ~90%+ with base FP16 model

Model Files

.
├── model.safetensors         # Quantized model weights (17GB)
├── config.json            # Model configuration
├── tokenizer.json       # Tokenizer data
├── tokenizer_config.json
├── generation_config.json
├── chat_template.jinja
├── .gitattributes
├── LICENSE
└── README.md

Quantization Details

The model was quantized using the following configuration:

from transformers import BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # Normal Float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,      # Double quantization for size reduction
)

Technical Details

• Quantization Type: NF4 (Normal Float)
• Double Quantization: Enabled (~5% additional size reduction)
• Compute Precision: bfloat16
• Activation Precision: float16
• Quantization Method: bitsandbytes

Quantization Process & Known Issues

The quantization was performed using AutoModelForCausalLM which caused the following:

1. Architecture Change: Original model uses Qwen3_5ForConditionalGeneration (multimodal). The quantized config was changed to Qwen3_5ForCausalLM (text-only). This has been manually fixed in config.json.

2. Vision Capabilities Lost: The original Qwen3.6-27B supports image+text input. The quantized version is text-only as vision components are not preserved by bitsandbytes quantization.

3. model_type Change: The quantization changed model_type from qwen3_5 to qwen3_5_text. This has been manually fixed in config.json.

[!IMPORTANT] If you need multimodal capabilities, use the original model: Qwen/Qwen3.6-27B

Features

• ✓ Agentic coding enhanced
• ✓ Extended context (up to 1M tokens)
• ✓ Thinking/reasoning mode
• ✓ Tool calling
• ✓ Multilingual support
• ✓ Code generation
• ✓ Debugging assistance
• ⚠ Text-Only: No vision/multimodal (lost during quantization)

Usage in AI Agents

This model works well with popular agent frameworks:

For 16GB GPUs (use CPU offload)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    max_memory={0: "14GB", "cpu": "30GB"},  # Offload to CPU for 16GB cards
    torch_dtype=torch.bfloat16,
)

LangChain

from langchain_community.llms import HuggingFacePipeline
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="/path/to/Qwen3.6-27B-4bit",
    device_map="auto",
    token="your_token"
)

llm = HuggingFacePipeline(pipeline=pipe)

LiteLLM

from litellm import litellm

response = litellm.completion(
    model="huggingface/Qwen3.6-27B-4bit",
    messages=[{"role": "user", "content": "Write a hello world program"}],
    api_key="your_token"
)

API Reference

Generation Parameters

# Recommended settings for code generation
generation_kwargs = {
    "max_new_tokens": 512,
    "temperature": 0.2,
    "do_sample": False,  # Use False for deterministic output
    "top_p": 0.95,
    "top_k": 20,
}

outputs = model.generate(**inputs, **generation_kwargs)

License

Apache 2.0 - See LICENSE for full license text.

Related Models

• Qwen3.6-27B (Original)
• Qwen3.5
• Qwen2.5-Coder

Support

• Hugging Face
• Documentation
• Discord

---

<div align="center"> <p>Built with 🤗 Transformers + bitsandbytes</p> <p>Quantized: April 2026</p> </div>