Downloads · 30 days
117
12% of all-time downloads
GaelicThunder/Qwen3.5-122B-A10B-heretic-hybrid-int4fp8
Qwen3.5-122B-A10B-heretic-hybrid-int4fp8 is a text generation model from GaelicThunder. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Uncensored Qwen3.5-122B-A10B (MoE, 122B total / ~10B active) quantized as a hybrid INT4 (MoE experts) + FP8 (shared-expert / dense) checkpoint with working MTP speculative decoding, tuned to run on a single NVIDIA DGX…
Downloads · 30 days
117
12% of all-time downloads
All-time downloads
977
Public
Parameters
20.7B
71.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors71.7 GB · 100%
How the weights are stored.
I3215.2B · 73%
From the Hugging Face model README
Uncensored Qwen3.5-122B-A10B (MoE, 122B total / ~10B active) quantized as a hybrid INT4 (MoE experts)
trohrbaugh/Qwen3.5-122B-A10B-heretic (KL≈0.09)model_free RTN; shared-expert → FP8 E4M3 (block-128) from Qwen/Qwen3.5-122B-A10B-FP8OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4)max-num-seqs 1)| Config | tok/s |
|---|---|
MTP num_spec=2 | 44.8 |
MTP num_spec=3 | 44.5 |
| no MTP | 36.6 |
Needs a vLLM built for sm_121 with the hybrid-FP8 dispatch patch — see
albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4
(image vllm-qwen35-v2). Also loads on stock vLLM as plain INT4 (slower, no FP8 shared-expert).
vllm serve GaelicThunder/Qwen3.5-122B-A10B-heretic-hybrid-int4fp8 \
--max-model-len 262144 --gpu-memory-utilization 0.75 --trust-remote-code \
--load-format fastsafetensors \
--enable-chunked-prefill --max-num-batched-tokens 8192 --max-num-seqs 1 \
--reasoning-parser qwen3 --attention-backend FLASHINFER \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--enable-auto-tool-choice --tool-call-parser qwen3_xml
config.json sets quantization_config.block_name_to_quantize = "model.language_model.layers".
This is required: it scopes quantization to the main transformer blocks so the MTP module
(mtp.*) and vision tower stay unquantized. Without it, vLLM's qwen3_5_mtp loader raises
KeyError: 'layers.0.mlp.experts.w2_weight'.
Full recipe + caveats + scripts: https://github.com/GaelicThunder/heretic-122b-dgx-spark
Apache-2.0 (inherited from Qwen/Qwen3.5-122B-A10B — verify). Abliteration by trohrbaugh.
This model is uncensored and will comply with requests a safety-tuned model would refuse.
You are responsible for lawful, ethical use.