Qwen3.6-35B-A3B-MTP-APEX-I-NanoPlus-GGUF

<a id="quick-navigation"></a>Quick Navigation Index

<a href="https://ko-fi.com/isvalorum"><img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support on Ko-fi" height="38"></a>

Independently computed using dedicated cloud compute clusters, funded 100% out-of-pocket.

If you consider these releases to be of high quality and they help you in your day-to-day workflow, consider supporting on Ko-fi. Every contribution directly sustains future cloud compute hours and helps support my university studies.

1. Optimization History & Transparency Notice
2. Independent Benchmark of the APEX-I-MiniPlus Family (Occamy V2 Reference)
3. Empirical Benchmarks & Fidelity Verification
4. Quality Spectrum: APEX-I-NanoPlus vs. Standard Flat Quantizations
5. Model Files & Technical Specifications
6. Surgical Tensor Quantization Map (Audited from GGUF)
7. Hardware Throughput & Offload Benchmarks (RTX 30 / 40 / 50 & RAM Streaming)
8. The 16GB & 24GB Advantage: Massive Context in VRAM!
9. Recommended Configuration & Setup
10. 1. llama-cli with Vision Projector
11. 2. High-Performance OpenAI-Compatible Server
12. Recommended Generation Parameters (Qwen Official)
13. CRITICAL: Coding Syntax & Repeat Penalty Advisory (Preventing Character Swapping)
14. Hardened Agentic Chat Template & Reasoning Effort
15. Optional Support

Qwen3.6-35B-A3B-MTP APEX-I-NanoPlus GGUF

The Next-Generation Frontier MoE · Extreme 13GB Footprint · Fast System RAM Streaming & Massive Context on 16GB VRAM

[!IMPORTANT]

THE DEFINITIVE SPECIFICATION IN THE approx. 13 GB CEILING

This APEX-I-NanoPlus release marks the official debut of our specialized tensor-by-tensor architectural configuration for sparse Mixture-of-Experts quantization within an extreme approx. 13 GB envelope. Its 40 main layers, 256 micro-experts, and integrated blk.40 MTP block are allocated tensor by tensor to maximize reasoning precision, preserve routing behavior, and prevent avoidable CPU dequantization stalls during hybrid and system-RAM inference.

[!TIP]

🏆 EMPIRICAL BENCHMARK & QUALITY COMPARISON

Quantization Specification	File Size (Disk)	Memory Footprint (RAM/VRAM)	Average BPW	WikiText-2 Perplexity	Quality Tier Equivalent
Unquantized BF16 Base	approx. 71.05 GB	approx. 66.18 GiB	16.00 BPW	approx. 5.32	Full Precision Baseline
APEX-I-MiniPlus V2.1	15.23 GB	14.18 GiB	3.43 BPW	5.3693 ± 0.12528 (+0.0493 / +0.93%)	Q5_K_L tier (bordering Q6_K)
APEX-I-NanoPlus (NEW)	13.03 GB	12.14 GiB	2.93 BPW	5.5244 ± 0.12916 (+0.2044 / +3.84%)	Solid Q4_K_M / Q4_K_L Tier
Generic Community IQ2_S	approx. 12.2 GB	approx. 11.4 GiB	2.56 BPW	> 5.85 (Degraded)	Unstable / Syntax Spikes

Looking for higher precision? APEX-I-MiniPlus V2.1 offers the full 15.23 GB (3.43 BPW) release of this Qwen3.6 family, delivering Q5_K_L tier (bordering Q6_K) fidelity with integrated MTP support.

ARC-Challenge (0-shot, 1,172 questions): approx. 95.75%.

• Q4_K_L Tier in Reasoning & Routing: all 80 main-model routers remain uncompressed F32, the two integrated MTP routers remain BF16, and the output head is protected in Q6_K.
• Solid Q4_K_M Tier in Language Modeling: WikiText-2 perplexity preserves 4-bit distributional fidelity across standard generation in an agile approx. 13.03 GB footprint.

Routing: all 80 main-model gate_inp / gate_inp_shexp tensors remain in uncompressed F32; the two integrated MTP routers in blk.40 are stored in BF16.

[!WARNING]

DO NOT CONFUSE APEX-I-NANOPLUS WITH GENERIC COMMUNITY SUB-3-BIT QUANTS!

Regardless of release version, NEVER confuse handcrafted APEX-I-NanoPlus builds with generic community sub-3-bit releases:

• Generic Community IQ2_S / IQ2_XXS: Uniformly crushes all core MoE experts down to aggressive 2-bit codebooks without importance calibration, leaves the sensitive token output head unarmored at 3-bit, and compresses attention projections. In deep reasoning models, this triggers severe perplexity spikes, syntax errors, and broken code brackets.
• Handcrafted APEX-I-NanoPlus: Applies a surgical tensor-by-tensor architecture that keeps the 80 main-model routing matrices in uncompressed F32, retains the two integrated MTP routers in BF16, armors the token output head in high-precision Q6_K, safeguards attention gates in Q8_0, fortifies the critical MoE down-projection residual stream (ffn_down_exps) in IQ3_XXS (3.06 bpw), and restricts 2-bit compression strictly to redundant gating/up projections guided by the official imatrix.

[!TIP]

SYSTEM RAM INFERENCE: FULL OR PARTIAL

This APEX-I-NanoPlus release is designed for full or partial system-RAM inference. Depending on the processor, memory bandwidth, and DDR4/DDR5 configuration, generation can range from 20 to 45 tok/s. With partial GPU offload, systems that cannot fit 128K or more context entirely in VRAM can place the remaining model and context load in system RAM, maintaining stable, responsive generation at longer context lengths.

---

<a id="toc-01"></a>

Optimization History & Transparency Notice

We maintain our previous releases publicly as a transparent engineering record of continuous optimization. Below is the exact evolutionary roadmap of our architectures:

[!TIP]

Deployment & System Architecture Guide

• Full GPU VRAM Offload (16GB+ VRAM, -ngl 99): Effortless full offload with native 32K–64K context support on 16GB cards (RTX 4080 / RTX 4070 Ti Super), and native 256K context on 24GB workstations (RTX 3090 / 4090 / 5090).
• System RAM Streaming Specialist (DDR4/DDR5 & Massive Context): Specially engineered to run either partially or entirely out of system RAM across large or full context windows. By utilizing linear SIMD-optimized Q4_K attention projections and preserving critical down-projections in IQ3_XXS, AVX2 CPU dequantization stalls are eliminated. Depending on your processor architecture and memory bandwidth (dual-channel DDR4 or high-speed DDR5 6000+ MT/s), streaming generation in system RAM can approach speeds remarkably close to full VRAM execution.

Explore our official collection: APEX-I-NanoPlus Collection.

---

[!IMPORTANT]

EXPLORE THE ESTABLISHED 35B MoE MINIPLUS LINEUP

These are complementary APEX-I-MiniPlus V2.1 releases, not alternate downloads of the same model. Each receives the same tensor-by-tensor approach, integrated MTP where supported, and a design suitable for full or partial system-RAM inference. Choose the model whose native strengths best fit the work you want to do:

• Qwen3.6-35B-A3B MTP APEX-I-MiniPlus-V2.1 — a versatile frontier MoE for broad reasoning, multilingual work, agents, tool use, and multimodal tasks.
  • Best for: General reasoning, agent workflows, tool calling, and flexible multimodal use.
• Qwen3.6-35B-A3B MTP APEX-I-MiniPlus-V2.1 Abliterated — the V2.1 refusal-ablated Qwen3.6 edition for users who deliberately prefer reduced refusal behavior.
  • Best for: Workflows where an abliterated Qwen3.6 variant is explicitly desired.
• Ornith 1.5 APEX-I-MiniPlus-V2.1 — a software-engineering-focused MoE designed for repository-scale coding and autonomous engineering agents.
  • Best for: Repository-scale development, multi-file code changes, and software-engineering agents.
• Tiel Coder APEX-I-MiniPlus-V2.1 — a specialist coding MoE tuned for agentic programming, iterative tool use, and implementation-heavy work.
  • Best for: Focused coding sessions, iterative debugging, and tool-driven implementation.

These remain distinct model families and editions with their own behavior and empirical results. Pick by workload and intended alignment behavior rather than treating them as interchangeable quantization variants.

---

<a id="independent-benchmark"></a> <a id="toc-02"></a>

🏅 Independent Benchmark of the APEX-I-MiniPlus Family (Occamy V2 Reference)

[!NOTE] External report: zephel01 independently benchmarked Occamy V2. The benchmark was performed on Occamy-1.0 APEX-I-MiniPlus V2, not on this specific NanoPlus release. It is included as independent context for the broader tensor-by-tensor MiniPlus approach.

The related APEX-I-MiniPlus quantization architecture has an independent evaluation by Japanese AI researcher and evaluator zephel01 (CoolZero) on an NVIDIA RTX 5090 (32GB) workstation running llama.cpp CUDA with FlashAttention.

The evaluation covered 348 unseeded trials on SWE-bench-style multi-file Python bug-fixing tasks with hidden pytest suites (llmbench):

• L6 Multi-File Code Generation (60 tasks):
  • 32K context: 93.3% resolved (46/60 tasks passed 5/5 consecutive trials).
  • 65K context: 90.0% resolved (45/60 tasks passed 5/5 consecutive trials).
• Context scaling on the tested Occamy V2 build: 14.6 GB at 32K and 15.1 GB at 65K.

These findings belong to Occamy V2 and should not be read as a direct NanoPlus benchmark. NanoPlus-specific fidelity is documented by its own WikiText-2 result below.

<a id="model-specifications"></a> <a id="empirical-benchmarks"></a> <a id="toc-03"></a>

Empirical Benchmarks & Fidelity Verification

The comparison table near the top consolidates the model-specific BF16 baseline, final GGUF PPL, delta, published main-file size, BPW, and fidelity tier. The routing treatment is preserved in the note directly beneath it.

<a id="quality-spectrum"></a> <a id="toc-04"></a>

Quality Spectrum: APEX-I-NanoPlus vs. Standard Flat Quantizations

How the handcrafted APEX-I-NanoPlus architecture compares against standard flat quantizations in llama.cpp on 35B Mixture-of-Experts architectures:

---

<a id="toc-05"></a>

Model Files & Technical Specifications

• Base Model: Qwen/Qwen3.6-35B-A3B
• Parameters: 35.2B total (approx. 2.6B to 3.2B active per token)
• Architecture: 40 main layers, 256 micro-experts (8 active per token) + hybrid linear attention / DeltaNet recurrent layers + integrated blk.40 MTP block
• Context Length: 262,144 tokens (native 256K)

---

<a id="tensor-map"></a> <a id="toc-06"></a>

Surgical Tensor Quantization Map (Audited from GGUF)

The exact tensor breakdown below has been verified directly from the compiled binary weights:

---

<a id="throughput-projections"></a> <a id="toc-07"></a>

Hardware Throughput & Offload Benchmarks (RTX 30 / 40 / 50 & RAM Streaming)

Hardware guidance for compatible llama.cpp-style runtimes:

• Aggressive Hybrid Offload Profile: Hybrid offload supports reasoning-enabled generation with limited VRAM while the remaining model weights stream from system RAM.

[!NOTE]

Empirical Testbed Architecture & Desktop/Server Scaling

• Empirical Benchmark Hardware: The hybrid offload and system RAM streaming behavior documented above was measured on a consumer laptop powered by an Intel 12th Gen Alder Lake architecture featuring a hybrid design of Performance Cores (P-Cores) and Efficient Cores (E-Cores) paired with dual-channel system RAM and constrained laptop power/thermal envelopes.
• Thread Scheduling & E-Core Contention: In hybrid architectures like Alder Lake, OS thread scheduling across background E-Cores and lower single-core mobile power limits introduce memory bandwidth and thread synchronization overhead during CPU dequantization.
• Dramatic Scaling on Higher-End Processors: When running on desktop or server processors (such as modern AMD Ryzen 7000 / 9000 Zen 4/5 series or high-TDP Intel desktop platforms with dedicated performance cores, large L3 caches, and high-bandwidth dual- or quad-channel DDR5 running at 6000+ MT/s), streaming generation speeds and prefill throughput will scale dramatically higher, substantially exceeding these measured mobile numbers.

---

<a id="context-scaling"></a> <a id="toc-08"></a>

The 16GB & 24GB Advantage: Massive Context in VRAM!

Qwen3.6-35B-A3B APEX-I-NanoPlus fits native context on 16GB GPUs and the full 256K window on 24GB GPUs:

Note: Leaves comfortable headroom for display drivers and compute buffers on standard 16GB and 24GB GPUs.

[!TIP]

💡 16GB GPU Verification (Single Stream / Desktop)

Single-stream desktop inference with FlashAttention consumes approx. 14.5 GB at 32,768 ctx and approx. 15.0 GB at 65,536 ctx (-ctk q8_0 -ctv q8_0 -fa on). This confirms that 32K to 64K context runs completely in VRAM on 16GB cards (RTX 4080 / RTX 4070 Ti Super / RTX 5080) without system RAM offload!

---

<a id="recommended-setup"></a> <a id="toc-09"></a>

Recommended Configuration & Setup

<a id="toc-10"></a>

1. llama-cli with Vision Projector

llama-cli \
  -m Qwen3.6-35B-A3B.APEX-I-NanoPlus.gguf \
  --mmproj mmproj-Q8_0.gguf \
  -ngl 99 \
  -c 32768 \
  --temp 0.6 \
  -p "Explain the implementation of a thread-safe LRU cache in Rust."

<a id="toc-11"></a>

2. High-Performance OpenAI-Compatible Server

llama-server.exe \
  -m Qwen3.6-35B-A3B.APEX-I-NanoPlus.gguf \
  --mmproj mmproj-Q8_0.gguf \
  -ngl 99 \
  --port 8080 \
  --parallel 4 \
  --flash-attn on \
  --fit on \
  -c 65536 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

---

<a id="generation-parameters"></a> <a id="toc-12"></a>

⚙️ Recommended Generation Parameters (Qwen Official)

Sampling metadata recorded from Qwen/Qwen3.6-35B-A3B in the completed build:

---

[!IMPORTANT] <a id="quantization-fidelity"></a>

🔍 Model Inherent Behavior vs. Quantization Fidelity Notice

Any behavioral nuances, stylistic tendencies, domain-specific habits, or zero-shot edge-case oversights stem entirely from the original unquantized checkpoint weights and fine-tuning distribution, NOT from the APEX-I quantization process. Handcrafted APEX-I-NanoPlus preserves the 80 main-model routing matrices in uncompressed F32, retains the two integrated MTP routers in BF16, armors the token output head in Q6_K, safeguards attention gates in Q8_0, and protects the down-projection residual stream in IQ3_XXS. Empirical verification records the final GGUF perplexity at 5.5244 ± 0.12916.

<a id="toc-coding-advisory"></a> <a id="coding-advisory"></a>

[!IMPORTANT]

CRITICAL ADVISORY FOR CODING WORKFLOWS: PREVENTING SYNTAX & TOKEN SWAPPING

In programming code, brackets ({, }), assignment operators (=), and indentation whitespace repeat constantly across multi-line structures.

Common Issue: Many local frontends (such as LM Studio defaults, Ollama, or web interfaces) ship with repeat_penalty set to 1.1 or 1.15. While this prevents loops in creative prose, applying repeat penalties to code artificially penalizes necessary syntax tokens. When the logit of { drops, the model is forced to emit the next closest mathematical token (= or [), resulting in character swapping or dropped/doubled whitespace.

Verified Upstream Behavior: This self-correcting behavior (where the model notices the mistake in its thinking loop but repeats the substitution) is documented on official upstream base checkpoints and Q8 builds (Ornith Discussion #34 and Discussion #22). It is completely eliminated by proper sampling configuration:

1. Disable Repeat Penalties (Required for Code):
   • repeat_penalty: 1.0 (strictly disabled)
   • presence_penalty: 0.0
   • frequency_penalty: 0.0
2. Calibrate Samplers:
   • temperature: 0.60 (or 0.20 - 0.30 for strict, deterministic code syntax)
   • min_p: 0.05 (prunes low-probability noise tokens effectively)
   • top_p: 0.95
   • top_k: 20
3. Native Jinja Formatting: Always pass the --jinja flag so the tokenizer handles leading-space BPE tokens ( { vs {, = vs =) cleanly.

<a id="toc-chat-template"></a> <a id="chat-template"></a>

[!TIP]

HARDENED AGENTIC CHAT TEMPLATE (JINJA)

An optimized chat_template.jinja is included at the root of this repository. It hardens agent workflows and multi-turn stability:

1. Native reasoning_effort Multi-Level Control:
   • low / minimal: Keeps internal thinking concise and focused strictly on immediate execution steps to minimize latency in automated loops.
   • medium (default): Balanced, structured reasoning process with standard analytical depth.
   • high / xhigh: Guides the model to formulate a clear implementation plan upfront before generating code, avoiding circular self-doubt loops.
   • none / off: Closes the thinking block immediately (<think>\n\n</think>) when reasoning is disabled.
2. Tool-Calling Safeguard (Anti-Premature Stop): Prevents the model from terminating a turn (<|im_end|>) at a colon or action declaration prior to outputting <tool_call>.
3. Multi-Turn Thinking Memory: Preserves historical <think> blocks across turns by default, preventing context distribution drift in 78K+ token runs.

Usage with llama-server:

llama-server -m Model.gguf --chat-template-file chat_template.jinja --reasoning-effort medium

<a id="toc-13"></a>

7. Community Compute Fund & Student Support

<a href="https://ko-fi.com/isvalorum"><img src="https://huggingface.co/spaces/IsValorum/MiniPlus-NanoPlus-Requests/resolve/main/assets/dance-gold-ship.gif" alt="Gold Ship dancing" width="128" align="right"></a>

All IsValorum quantizations will always remain completely free and open to the public without paywalls.

However, cloud GPU compute is expensive and funded entirely out-of-pocket. If you find these handcrafted builds to be of high quality and they benefit your day-to-day workflow, you can support future Day-0 releases on Ko-fi. Every contribution directly funds the cloud compute hours required for new model runs and helps support my university education:

<a href="https://ko-fi.com/isvalorum"><img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support on Ko-fi" height="38"></a>

(When supporting on Ko-fi, feel free to leave a note with your Hugging Face handle and the specific model you would like prioritized).

<div style="clear: both;"></div>