Downloads · 30 days
24
3% of all-time downloads
sakamakismile/Huihui-Qwen3.5-27B-abliterated-NVFP4
Huihui-Qwen3.5-27B-abliterated-NVFP4 is a image-text-to-text model from sakamakismile. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
NVFP4 quantized version of huihui-ai/Huihui-Qwen3.5-27B-abliterated — an abliterated (uncensored) Qwen 3.5 27B dense model with multimodal capability and MTP (Multi-Token Prediction) support.
Downloads · 30 days
24
3% of all-time downloads
All-time downloads
937
Public
Parameters
16.7B
20.6 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors20.6 GB · 100%
How the weights are stored.
U812.2B · 73%
From the Hugging Face model README
NVFP4 quantized version of huihui-ai/Huihui-Qwen3.5-27B-abliterated — an abliterated (uncensored) Qwen 3.5 27B dense model with multimodal capability and MTP (Multi-Token Prediction) support.
~52 GB → 20.6 GB with high-quality 512-sample calibration. Fits on a single NVIDIA Blackwell GPU.
--enable-auto-tool-choice --tool-call-parser qwen3_xml| Base model | huihui-ai/Huihui-Qwen3.5-27B-abliterated |
| Architecture | Qwen 3.5 Dense — 27B parameters, 64 layers |
| Quantization | NVFP4 W4A4 (weights FP4, activations FP4, scales FP8) |
| Format | compressed-tensors (native vLLM support) |
| Tool | vllm-project/llm-compressor (main) |
| Calibration | 512 samples, neuralmagic/calibration, seq_len=4096 |
| Size | 20.6 GB |
| Max context | 262,144 tokens |
| Requires | NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130) |
vllm serve Lna-Lab/Huihui-Qwen3.5-27B-abliterated-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3
vllm serve Lna-Lab/Huihui-Qwen3.5-27B-abliterated-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--kv-cache-dtype fp8
vllm serve Lna-Lab/Huihui-Qwen3.5-27B-abliterated-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
docker run --gpus '"device=0"' -p 8016:8016 \
-v /path/to/model:/models/current:ro \
--shm-size 16gb \
-e VLLM_NVFP4_GEMM_BACKEND=marlin \
vllm/vllm-openai:cu130-nightly \
vllm serve /models/current --port 8016 --max-model-len 32768 \
--reasoning-parser qwen3
from vllm import LLM, SamplingParams
llm = LLM(
model="Lna-Lab/Huihui-Qwen3.5-27B-abliterated-NVFP4",
max_model_len=32768,
gpu_memory_utilization=0.90,
)
output = llm.generate(
["Implement a thread-safe LRU cache in Python with O(1) operations."],
SamplingParams(max_tokens=1024, temperature=0.3),
)
print(output[0].outputs[0].text)
Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).
| Test | Speed | Tokens | Result |
|---|---|---|---|
| English (system design) | 59.2 tok/s | 512 | PASS |
| Code (async scheduler) | 59.2 tok/s | 512 | PASS |
| Math (Bayes' theorem) | 59.2 tok/s | 512 | PASS |
| Japanese (technical writing) | 59.0 tok/s | 512 | PASS |
Sustained throughput: ~59 tok/s (single GPU, post-warmup).
Note: This is a 27B dense model (all parameters active), so per-token speed is lower than MoE models like Gemma 4 26B-A4B (~130 tok/s with only 3.8B active). However, the reasoning depth per token is significantly higher.
recipe = QuantizationModifier(
targets=["Linear"],
ignore=["lm_head", "re:.*visual.*", "re:.*in_proj_a$", "re:.*in_proj_b$"],
scheme="NVFP4",
)
Following the proven recipe from lyf/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive-NVFP4.
Linear layers in the text modellm_head, visual encoder, linear attention projections (in_proj_a, in_proj_b), MTP headMTP tensors are grafted from the original BF16 checkpoint using save_mtp_tensors_to_checkpoint. This preserves the speculative decoding head at full precision, enabling ~3x speedup with num_speculative_tokens=1.
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import Qwen3_5ForConditionalGeneration, AutoProcessor, AutoTokenizer
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch
MODEL_ID = "huihui-ai/Huihui-Qwen3.5-27B-abliterated"
OUTPUT = "Huihui-Qwen3.5-27B-abliterated-NVFP4"
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
recipe = QuantizationModifier(
targets=["Linear"],
ignore=["lm_head", "re:.*visual.*", "re:.*in_proj_a$", "re:.*in_proj_b$"],
scheme="NVFP4",
)
ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:512]")
def preprocess(example):
messages = [
{"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
for m in example["messages"]
]
return processor.apply_chat_template(
messages, return_tensors="pt", padding=False, truncation=True,
max_length=4096, tokenize=True, add_special_tokens=False,
return_dict=True, add_generation_prompt=False,
)
ds = ds.map(preprocess, batched=False, remove_columns=ds.column_names)
def data_collator(batch):
assert len(batch) == 1
return {
key: (torch.tensor(value) if key != "pixel_values"
else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
for key, value in batch[0].items()
}
oneshot(
model=model, recipe=recipe, dataset=ds,
max_seq_length=4096, num_calibration_samples=512,
data_collator=data_collator,
)
model.save_pretrained(OUTPUT, save_compressed=True)
processor.save_pretrained(OUTPUT)
tokenizer.save_pretrained(OUTPUT)
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=OUTPUT)
| Package | Version |
|---|---|
| torch | 2.11.0+cu130 |
| transformers | 5.5.4 |
| llmcompressor | 0.1.dev (main @ 3084520) |
| compressed-tensors | 0.15.1a20260414 |
| CUDA | 13.0 |
--kv-cache-dtype fp8 to fit in 96 GB.If you find this model useful, please consider supporting huihui-ai:
bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge