Downloads · 30 days
74
4% of all-time downloads
jorge-erdb/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled-4bit-GGUF
Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled-4bit-GGUF is a image-text-to-text model from jorge-erdb. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
4-bit GGUF quantizations of Jongsim/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled, a Claude 4.6 Opus reasoning-distilled fine-tune of Jongsim/Qwen3.5-35B-A3B-heretic (SOMA+MPOA abliterated Qwen/Qwen3.5-35B-A3B).
Downloads · 30 days
74
4% of all-time downloads
All-time downloads
1.9K
Public
Repo size
40.9 GB
Likes
1
Public
Click a slice to open those files.
.gguf40.9 GB · 100%
From the Hugging Face model README
4-bit GGUF quantizations of Jongsim/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled, a Claude 4.6 Opus reasoning-distilled fine-tune of Jongsim/Qwen3.5-35B-A3B-heretic (SOMA+MPOA abliterated Qwen/Qwen3.5-35B-A3B).
| Detail | Value |
|---|---|
| Quant types | IQ4_NL, Q4_K_M |
| Quantized by | jorge-erdb |
| Method | llama.cpp with importance matrix |
| Importance matrix | bartowski's imatrix calibration dataset |
| Source model | Jongsim/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled (BF16) |
| Quant | Best for | Notes |
|---|---|---|
| IQ4_NL | CUDA / CPU | Non-linear, slightly better quality per bit |
| Q4_K_M | CUDA / CPU / Metal | Linear, best Metal compatibility |
pip install -U "huggingface_hub[cli]"
# IQ4_NL (non-linear, best for CUDA/CPU)
huggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled-4bit-GGUF --include "*IQ4_NL.gguf" --local-dir ./
# Q4_K_M (linear, Metal-friendly)
huggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-heretic-Opus-4.6-Distilled-4bit-GGUF --include "*Q4_K_M.gguf" --local-dir ./
[!Important]
Apple Metal Backend Warning
IQ4_NL is a non-linear quantization format. It performs sub-optimally on Apple's Metal backend due to the lack of native support for non-linear dequantization kernels. If you are running on an Apple Silicon Mac with GPU offloading via Metal, you will likely experience:
- Slower inference compared to linear quants of similar size (e.g., Q4_K_M)
- No speed benefit from the ARM weight repacking that IQ4_NL supports on CPU
If you're on Apple Metal, use the Q4_K_M quant from this repo instead. For higher precision options, see Jongsim's repo.
A reasoning-enhanced, abliterated version of Qwen3.5-35B-A3B (35B total / 3B active parameters, Mixture of Experts). This model was built in two stages: first, censorship removal via directional ablation using Heretic, then supervised fine-tuning on high-quality Chain-of-Thought reasoning traces distilled from Claude 4.6 Opus.
The model produces structured reasoning within <think>...</think> tags before delivering final responses. All weights are in bf16 precision.
This model is a fine-tuned derivative of Jongsim/Qwen3.5-35B-A3B-heretic, which itself is an abliterated (decensored) version of Qwen/Qwen3.5-35B-A3B.
The primary objective is to inject high-density structured reasoning capability from Claude 4.6 Opus while preserving the uncensored nature of the abliterated base model. Through SFT on curated reasoning distillation data, the model learns to decompose complex problems into sequential steps within a dedicated thinking block before generating the final answer.
| Property | Value |
|---|---|
| Architecture | Qwen3.5 MoE (Gated DeltaNet + Gated Attention + MoE) |
| Total Parameters | 35B |
| Active Parameters | 3B per token |
| Hidden Dimension | 2048 |
| Layers | 40 (10 repeating blocks of 3x DeltaNet-MoE + 1x Attention-MoE) |
| Experts | 256 total, 8 routed + 1 shared active |
| Expert Intermediate Dim | 512 |
| Context Length | 262,144 tokens (native) |
| Precision | bf16 |
| Vocabulary | 248,320 tokens |
Qwen/Qwen3.5-35B-A3B (original)
|
| Heretic v1.2.0 (SOMA + MPOA abliteration)
v
Jongsim/Qwen3.5-35B-A3B-heretic (abliterated base)
|
| Supervised Fine-Tuning (LoRA + Unsloth)
v
Jongsim/Qwen3.5-35B-A3B-heretic-Reasoning (this model)
The base model was processed with Heretic v1.2.0, an automated censorship removal tool that applies directional ablation optimized via Bayesian hyperparameter search (Optuna TPE).
Two techniques were combined:
| Parameter | Value |
|---|---|
| Method | SOMA + MPOA |
| Orthogonalize Direction | true |
| Row Normalization | full |
| Full Normalization LoRA Rank | 4 |
| Winsorization Quantile | 0.95 |
| SOM Grid | 4 x 4 (16 neurons) |
| SOM Iterations | 10,000 |
| SOM Learning Rate | 0.01 |
| SOM Sigma | 0.5 |
| SOM k (directions) | 4 |
| Optimization Trials | 200 (60 startup) |
| Selected Trial | Trial 84 / 200 |
| Good Prompts | mlabonne/harmless_alpaca (train[:400]) |
| Bad Prompts | mlabonne/harmful_behaviors (train[:400]) |
| Quantization | none (bf16) |
| Metric | Original | Abliterated |
|---|---|---|
| KL Divergence | 0 (reference) | 0.0638 |
| Refusals (out of 100) | 91 | 6 |
93.4% refusal reduction with minimal distribution shift (KL = 0.0638).
Inject structured Chain-of-Thought reasoning patterns from Claude 4.6 Opus into the abliterated model. The training enforces a strict output format where the model generates internal reasoning within <think> blocks before producing the final response.
train_on_responses_only — loss is calculated exclusively on assistant responses (both thinking trace and final answer), not on user prompts
<|im_start|>user\n<|im_start|>assistant\n<think><|im_start|> / <|im_end|>)| Parameter | Value |
|---|---|
| PEFT Method | LoRA |
| Rank (r) | 16 |
| Alpha | 32 (= 2 x rank) |
| Dropout | 0.0 |
| Bias | none |
| Target Modules (Attention) | q_proj, k_proj, v_proj, o_proj |
| Target Modules (FFN) | gate_proj, up_proj, down_proj |
| Target Modules (MoE) | gate_up_proj |
| Gradient Checkpointing | unsloth mode |
| Parameter | Value |
|---|---|
| Max Sequence Length | 2,048 |
| Per-Device Batch Size | 1 |
| Gradient Accumulation Steps | 8 |
| Effective Batch Size | 8 |
| Number of Epochs | 5 |
| Total Training Steps | 1,995 |
| Learning Rate | 2e-4 |
| LR Scheduler | Linear decay |
| Warmup Steps | 5 |
| Optimizer | AdamW 8-bit |
| Weight Decay | 0.001 |
| Precision | bf16 |
| Seed | 3407 |
| Total FLOPs | 3.56 x 10^18 |
Three publicly available reasoning distillation datasets were combined, shuffled (seed=42), and used for training:
| Dataset | Samples | Description |
|---|---|---|
| nohurry/Opus-4.6-Reasoning-3000x-filtered | ~2,308 | Filtered reasoning trajectories from Claude 4.6 Opus. Each sample contains a problem, a detailed thinking trace, and a final solution. |
| TeichAI/claude-4.5-opus-high-reasoning-250x | ~250 | High-intensity structured reasoning instances from Claude 4.5 Opus multi-turn conversations. |
| Jackrong/Qwen3.5-reasoning-700x | ~633 | Curated reasoning samples in both conversation and instruction format, designed for step-by-step problem solving diversity. |
| Total | ~3,191 | Combined after filtering empty/invalid rows. |
| Epoch | Avg Loss | Steps |
|---|---|---|
| 1 | 0.4299 | 79 - 399 |
| 2 | 0.3729 | 400 - 798 |
| 3 | 0.3359 | 799 - 1197 |
| 4 | 0.3059 | 1198 - 1596 |
| 5 | 0.2958 | 1597 - 1995 |
Training loss decreased monotonically from 0.4299 to 0.2958 across 5 epochs, indicating stable convergence without overfitting signs at the loss level.
The best checkpoint was selected based on GSM8K accuracy (50 samples). All checkpoints were evaluated in isolated subprocesses to prevent GPU memory leaks from Unsloth's model patching.
| Checkpoint | Epoch | GSM8K Accuracy |
|---|---|---|
| checkpoint-1200 | 3.0 | 8.0% (4/50) |
| checkpoint-1400 | 3.5 | 10.0% (5/50) |
| checkpoint-1596 | 4.0 | 10.0% (5/50) |
| checkpoint-1995 | 5.0 | 12.0% (6/50) |
checkpoint-1995 (epoch 5) was selected and merged into bf16 for the final release.
Note: GSM8K measures narrow arithmetic reasoning and does not fully reflect the model's broader reasoning capabilities (code generation, logical analysis, multi-step planning) which are the primary targets of the distillation training.
| Component | Value |
|---|---|
| Hardware | NVIDIA DGX Spark |
| GPU | NVIDIA GB10 (128GB unified memory) |
| Compute Capability | sm121 |
| Architecture | aarch64 |
| CUDA | 13.0 |
| PyTorch | 2.9.1a0 |
| Transformers | 5.2.0 |
| Unsloth | 2026.3.3 |
| TRL | 0.24.0 |
| PEFT | 0.18.1 |
| Datasets | 4.3.0 |
| Tokenizers | 0.22.2 |
flash_attn package is fully removed to prevent FATAL errors on sm121.torch.compile / TorchInductor is disabled due to Triton ptxas compatibility issues.This model uses the standard Qwen3.5 chat template. It operates in thinking mode by default.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Jongsim/Qwen3.5-35B-A3B-heretic-Reasoning"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
messages = [
{"role": "user", "content": "Explain the difference between TCP and UDP, and when you would choose one over the other."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=8192, temperature=0.7, top_p=0.8, top_k=20)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)
| Mode | temperature | top_p | top_k | presence_penalty |
|---|---|---|---|---|
| Thinking (general) | 1.0 | 0.95 | 20 | 1.5 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 |
| Non-thinking (general) | 0.7 | 0.8 | 20 | 1.5 |
The model produces output in the following structure:
<think>
Let me analyze this problem step by step.
1. First, I need to identify the core question being asked.
2. Then, I'll consider the relevant constraints and conditions.
3. Next, I'll work through the logic systematically.
4. Finally, I'll verify my reasoning for consistency.
[detailed reasoning follows...]
</think>
[final answer here]
This structured thinking pattern, distilled from Claude 4.6 Opus interactions, reduces redundant cognitive loops while preserving deep analytical capacity.
@misc{jongsim_qwen35_heretic_reasoning,
title = {Qwen3.5-35B-A3B-heretic-Reasoning},
author = {Jongsim},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Jongsim/Qwen3.5-35B-A3B-heretic-Reasoning}}
}