Downloads · 30 days
543
1% of all-time downloads
sakamakismile/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-NVFP4
Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-NVFP4 is a image-text-to-text model from sakamakismile. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
NVFP4 quantized version of huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated — Claude 4.7 Opus distilled, abliterated (uncensored) Qwen 3.6 MoE with 256 experts and 3B active parameters.
Downloads · 30 days
543
1% of all-time downloads
All-time downloads
58.7K
Public
Repo size
21.9 GB
Likes
21
Public
Click a slice to open those files.
.safetensors21.9 GB · 100%
From the Hugging Face model README
NVFP4 quantized version of huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated — Claude 4.7 Opus distilled, abliterated (uncensored) Qwen 3.6 MoE with 256 experts and 3B active parameters.
67 GB → 21.9 GB. Single NVIDIA Blackwell GPU. 182 tok/s. 256K context. VLM. Uncensored.
Claude 4.7 Opus intelligence distilled into a locally runnable MoE, with abliteration for unrestricted research use:
| Base model | huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated |
| Architecture | Qwen3.5 MoE — 35B total, 3B active, 256 experts (8 routed + 1 shared) |
| Quantization | NVFP4 W4A4 (weights FP4, activations FP4, scales FP8) |
| Format | compressed-tensors (native vLLM support) |
| Tool | vllm-project/llm-compressor (main) |
| Calibration | 512 samples, ultrachat_200k, seq_len=2048, moe_calibrate_all_experts=True |
| Size | 21.9 GB |
| Max context | 262,144 tokens (native) |
| Requires | NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130) |
vllm serve sakamakismile/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--kv-cache-dtype fp8
vllm serve sakamakismile/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--kv-cache-dtype fp8
docker run --gpus device=0 -p 8090:8090 \
-v /path/to/model:/models/current:ro \
--shm-size 16gb \
vllm/vllm-openai:cu130-nightly \
vllm serve /models/current --port 8090 --max-model-len 32768 \
--reasoning-parser qwen3 --kv-cache-dtype fp8
Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), 256K context, FP8 KV cache.
| Test | Tokens | Speed | Result |
|---|---|---|---|
| English (CAP theorem) | 256 | 182 tok/s | PASS |
| Japanese (量子コンピュータ) | 256 | 182 tok/s | PASS |
| Code (async scheduler) | 512 | 182 tok/s | PASS |
| Math (Bayes theorem) | 512 | 182 tok/s | PASS |
| Burst stability (×3) | 512 | 182-188 tok/s | PASS — stable |
| VLM (shape recognition) | 256 | — | PASS ✅ |
Sustained: ~182 tok/s (single GPU, 256K context).
| Context Length | VRAM | KV Cache |
|---|---|---|
| 262,144 (256K) | 95.6 GB | FP8 |
256K context fits on a single 96 GB Blackwell GPU with FP8 KV cache.
| Model | Speed | Link |
|---|---|---|
| Qwen3.6-35B-A3B (base) | 182 tok/s | sakamakismile/Qwen3.6-35B-A3B-NVFP4 |
| Huihui abliterated | 175 tok/s | sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 |
| Claude 4.6 Opus abliterated | 175 tok/s | sakamakismile/Huihui-Qwen3.6-35B-A3B-Claude-4.6-Opus-abliterated-NVFP4 |
recipe = QuantizationModifier(
targets="Linear",
scheme="NVFP4",
ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
)
moe_calibrate_all_experts=True — ensures all 256 experts receive calibration datafrom transformers import Qwen3_5MoeForConditionalGeneration, AutoProcessor, AutoTokenizer
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_ID = "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
recipe = QuantizationModifier(
targets="Linear", scheme="NVFP4",
ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
)
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:512]")
ds = ds.shuffle(seed=42)
def preprocess(example):
return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
ds = ds.map(preprocess)
def tokenize(sample):
return tokenizer(sample["text"], padding=False, max_length=2048,
truncation=True, add_special_tokens=False)
ds = ds.map(tokenize, remove_columns=ds.column_names)
oneshot(model=model, dataset=ds, recipe=recipe,
max_seq_length=2048, num_calibration_samples=512,
moe_calibrate_all_experts=True)
model.save_pretrained("output", save_compressed=True)
processor.save_pretrained("output")
tokenizer.save_pretrained("output")
| Package | Version |
|---|---|
| torch | 2.11.0+cu130 |
| transformers | 5.5.4 |
| llmcompressor | 0.1.dev (main) |
| compressed-tensors | 0.15.1a20260414 |
| CUDA | 13.0 |
--kv-cache-dtype fp8 for 2x KV capacity at no quality cost.bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge