Downloads · 30 days
106
2% of all-time downloads
luffydenolan/Qwen3.6-35B-A3B-APEX-IQuality-GGUF
Qwen3.6-35B-A3B-APEX-IQuality-GGUF is a machine learning model from luffydenolan. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for hermes. The card lists the license as apache-2.0.
Qwen3.6-35B-A3B quantized with APEX imatrix built from real Hermes agent session traces.
Downloads · 30 days
106
2% of all-time downloads
All-time downloads
4.6K
Public
Repo size
23.9 GB
Likes
10
Public
Click a slice to open those files.
.gguf23.7 GB · 100%
From the Hugging Face model README
Qwen3.6-35B-A3B quantized with APEX imatrix built from real Hermes agent session traces.
Most GGUF quants calibrate on generic text. This one calibrates on actual agentic workloads — tool calls, multi-turn reasoning, code generation, and task completions from production Hermes agent sessions. If you run local agents, the quantization importance weights reflect your actual inference distribution.
Built from Qwen/Qwen3.6-35B-A3B — a 35B hybrid MoE model with 256 experts, 8 active per token, 34.66B total parameters. Combines attention layers with Gated Delta Net SSM layers (full attention every 4th layer), trained to 262K context.
This GGUF applies APEX quantization with a custom imatrix built entirely from Hermes agent session traces — real multi-turn conversations including tool calls, reasoning chains, code generation, and agentic task completions. No generic wikitext. Verified for use with Hermes Agent on Apple Silicon (M1 Max).
Base Model: Qwen/Qwen3.6-35B-A3B
Calibration Dataset: Combined imatrix calibration
extract_hermes_traces.py with Qwen3.6 chat templateAPEX Quantization: mudler/apex-quant
TurboQuant backend: Custom Metal kernels for M-series Apple Silicon
This release: @luffydenolan
Qwen/Qwen3.6-35B-A3B base (34.66B params, ~65GB f16, ~34GB Q8_0)llama-imatrix on Q8_0:
-c 512, --chunks 200, --threads 10, -ngl 99Qwen3.6-35B-A3B is a hybrid MoE + SSM model:
Qwen3.6-35B-A3B-apex-iquality.gguf — Main model weights (21 GB)mmproj-F16-Qwen3.6-35B-A3B.gguf — Multimodal projection layer, vision + video (858 MB)llama-server \
-m Qwen3.6-35B-A3B-apex-iquality.gguf \
-ngl 99 \
-t 4 \
--ctx-size 132000 \
--cache-type-k q8_0 \
--cache-type-v turbo4 \
--flash-attn on \
--jinja \
--chat-template-file chat_template_qwen36.jinja \
--temp 0.7 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
llama-server \
-m Qwen3.6-35B-A3B-apex-iquality.gguf \
--mmproj mmproj-F16-Qwen3.6-35B-A3B.gguf \
-ngl 99 \
-t 4 \
--ctx-size 132000 \
--cache-type-k q8_0 \
--cache-type-v turbo4 \
--flash-attn on \
--jinja \
--chat-template-file chat_template_qwen36.jinja \
--temp 0.7 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
| Parameter | Value | Notes |
|---|---|---|
| temperature | 0.7 | |
| top-k | 20 | Qwen3.x default |
| top-p | 0.95 | |
| min-p | 0.0 | |
| presence-penalty | 0.0 | |
| repeat-penalty | 1.0 | |
| cache-type-k | q8_0 | Best attention accuracy on M-series |
| cache-type-v | turbo4 | Good compression, less sensitive than K |
| ctx-size | 132000 | ~half of trained 262K — fits 64GB with headroom |
from llama_cpp import Llama
llm = Llama(
model_path="Qwen3.6-35B-A3B-apex-iquality.gguf",
n_ctx=132000,
n_gpu_layers=99,
n_threads=4,
)
response = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}]
)
print(response["choices"][0]["message"]["content"])
Tested on Apple M1 Max (64GB unified memory), llama.cpp via Metal + TurboQuant backend, 4 threads.
| Test | Tokens/sec |
|---|---|
| Prompt processing — pp128 | 360.60 ± 13.95 |
| Prompt processing — pp512 | 620.68 ± 2.74 |
| Prompt processing — pp1024 | 611.77 ± 6.17 |
| Token generation — tg128 | 42.24 ± 0.04 |
| Token generation — tg512 | 41.87 ± 0.22 |
turbo4 KV cache requires TheTom's TurboQuant fork of llama.cpp.
https://github.com/TheTom/turboquant_plus/blob/main/docs/getting-started.md
| Component | Size |
|---|---|
| Model weights (APEX iQuality) | ~21 GB |
| KV cache (q8_0 K / turbo4 V, 10 attn layers) | ~2 GB |
| SSM recurrent state | ~0.5 GB |
| Metal buffers + overhead | ~3 GB |
| Total RSS | ~26.6 GB |
~37 GB headroom remaining on 64GB for OS + other processes.
Apache 2.0 (inherited from base model)
@misc{qwen36-35b-a3b-apex-iquality-gguf,
title = {Qwen3.6-35B-A3B-APEX-IQuality-GGUF},
author = {luffydenolan},
year = {2026},
url = {https://huggingface.co/luffydenolan/Qwen3.6-35B-A3B-APEX-IQuality-GGUF}
}