Qwopus3.6-27B-Fusion-BF16

<div align="center"> <img src="assets/hero-banner.png" alt="Qwopus3.6-27B Fusion — one 27B that reasons and codes, no skill lost" width="100%"> <h3>▶ <a href="https://huggingface.co/spaces/KyleHessling1/ORBITAL">PLAY <i>ORBITAL</i> — a game this model built, live in your browser</a> ◀</h3> </div>

⬛ Full-precision BF16 weights. This repo hosts the unquantized BF16 GGUF (~54.7 GB) — the reference weights every quant is derived from. Want a smaller file that runs on one GPU? The quantized GGUFs (Q3_K_M, Q4_K_M, Q5_K_M, Q6_K) live in the companion repo → KyleHessling1/Qwopus3.6-27B-Fusion-GGUF.

Research preview. This model attempts to combine the reasoning capability of Qwopus3.6-27B-v2 with the code-execution capability of Qwopus3.6-27B-Coder — in a single 27B model, without significant loss to either skill. In practice it behaves like a production model with both capabilities fused, but it has not yet been through a full rigorous evaluation. If you find issues, please reach out on X — @kylehessling1.

• Format: GGUF, BF16 full precision (~54.7 GB) — the source weights; needs ~55 GB of RAM/VRAM (multi-GPU or CPU/llama.cpp offload). For single-GPU use, grab a quantized GGUF from the companion repo.
• Architecture: qwen35 hybrid (linear-attention + periodic full-attention), 27B params, 64 layers.
• Native context: 262,144 tokens. Verified clean (needle + termination) to 60K.
• Modes: thinking (reasoning) on by default; strong agentic/coding behavior.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ TL;DR

Headline results (Q4_K_M, thinking-on unless noted):

Against the alternatives on the same astropy slice: Fusion 7/15 > equal-weight DARE-TIES 5/15 > DUS-38B frankenmerge 3/15.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Where it shines — the one-stop creative programming agent

Best used as a single, closed-loop agentic creative programmer. In a build → run → self-fix loop, Fusion is at its strongest as one agent that plans, writes, and debugs on its own — the reasoning⊕coding balance lets it close the loop by itself (see the ORBITAL / FLUX demos below, where it root-caused and fixed its own bugs from Playwright error reports).

This is a different niche than its parents. The Coder model is excellent for coding under active steering, or as a solid orchestration model driving other tools/agents. Fusion, by contrast, is the ideal one-stop creative builder — you hand it a goal and it carries the whole creative-programming loop end-to-end.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Why I chose to make this merge

Two finetunes of the same base diverge into complementary specialists: one gets better at multi-step reasoning, the other at writing code — but each pays for it. In particular the coding finetune becomes unstable in thinking mode (it loops / degrades at low temperature). The goal was a single model that keeps the reasoner's stability and the coder's skill.

My naive approaches, one of them eroneously proposed by Claude Fable, average the two, or run an equal-weight DARE-TIES merge from the common base. These alternatives underperformed and the geometry explains why.

▌ The lineage

Because the coder model descended from the v2 version, the coders additions can be evaluated against the v2 by analyzing the delta. Projecting the coding delta onto the reasoning delta reveals:

proj(ΔCoder · ΔV2 / |ΔV2|²)   ≈ 0.4  in early layers   →   ≈ 1.0  in late layers

That ramp to 1.0 means late layers of the coder already contain V2's full reasoning delta. And:

• The coding delta is ~1.8× the magnitude of the reasoning delta.
• In early layers the coding delta is anti-aligned with reasoning (cos < 0) — the coder partially reverted V2's early-layer reasoning/control representations. That anti-alignment is the measured source of the coder's thinking-mode instability.

An equal-weight merge (both task vectors added from the base) therefore commits two errors:

1. It double-counts V2's reasoning delta (the coder already contains it), and
2. It imports the full early-layer destabilizing coding delta at 100% strength.

▌ The fix

Because the coder sits "downstream" of V2, the correct anchor is V2 itself, not the shared base — this avoids the double-count. Then, instead of a constant fraction of the coding delta everywhere, inject it with a depth-increasing schedule:

• Early layers (α ≈ 0.12): inject little coding delta → protect V2's reasoning/control reps exactly where the coder damaged them.
• Late layers (α ≈ 0.48): inject more coding delta → add coding skill where the two parents already agree (proj → 1.0) and it's safe.

The shape of the α ramp is dictated by the geometry, not hand-tuning. Embeddings, LM head, norms, and the MTP head are identical across all three models (delta ≈ 0), so they're copied verbatim from V2 rather than merged.

<div align="center"> <img src="assets/geometry.png" alt="Weight geometry — coder descends from V2, its delta is 1.8x and anti-aligned in early layers; the alpha ramp injects coding 0.12 early to 0.48 late" width="100%"> </div>

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ How it was built

Merge. The parents use Qwen's hybrid attention schedule (most layers linear/DeltaNet; every 4th layer full softmax attention). Off-the-shelf mergekit cannot merge this architecture — it templates linear_attn.* onto every layer and errors on the full-attention layers. So the merge is a custom streaming task-vector routine operating directly on the (identical) tensor keys, immune to arch templating. The full, reproducible script ships in this repo as merge_layerweighted.py:

def alpha(L, a_early, a_late):
    return a_early + (a_late - a_early) * (L / (NLAYERS - 1))   # smooth linear ramp by depth

# per decoder tensor at layer L:
out = (v2 + alpha(L, 0.12, 0.48) * (coder - v2)).to(torch.bfloat16)
# embeddings / lm_head / norms / mtp / vision: copied from V2 verbatim

Quantize. The bf16 merge was converted to GGUF and quantized to Q4_K_M with llama.cpp (851 GGUF tensors, 16.5 GB).

Evaluate. Every candidate was gated at both temp 0.2 (a termination stress test) and temp 0.9 (the finetune operating point) — "clean only at high temp" is riding sampling entropy, not stability. This model is clean at both.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Demonstrations — driving a local agentic coding loop

Every artifact below was generated by this model running as the backend of an agentic coding loop (OpenCode / Claude Code), building single-file HTML/JS apps end-to-end, then verified with headless Playwright. What stood out isn't just that it writes polished games — it's that when the Playwright harness handed back error reports, the same model root-caused and fixed its own bugs with precise, correct diagnoses (excerpts below).

🚀 ORBITAL — physics space-arcade (thinking-on)

Setup: "Build a complete, physics-driven, visually stunning arcade space game as a single self-contained index.html — gravity is the weapon." Outcome: built and runs cleanly, zero console errors, real 2-body gravity, particle trails, wave HUD, and it even self-balance-tuned its own difficulty. 1,150 lines.

<div align="center"> <h3>▶ <a href="https://huggingface.co/spaces/KyleHessling1/ORBITAL">PLAY IT LIVE → huggingface.co/spaces/KyleHessling1/ORBITAL</a> ◀</h3> <i>(runs entirely in your browser — the exact single file this model wrote)</i> </div>

The same loop also produced three more complete, self-contained games — PRISM DEFENSE (neon tower defense, 1,074 lines), NEON SERPENT (synthwave Snake with smooth interpolated movement, 657 lines), and STARLIGHT SURVIVORS (survivors-style auto-shooter, 555 lines). Each was built, Playwright-tested, and where the harness surfaced a bug the model root-caused and fixed it itself (e.g. a tower-targeting bestDist = Infinity that should have been -Infinity; three weapon upgrades missing a fireTimer init that made fireTimer -= dt evaluate to NaN). The impressive part is the quality of the model's own debugging inside a build → error report → self-fix loop, not flawless one-shot output.

📋 FLUX — a full Trello-style Kanban dashboard (the strongest capability demo)

The most demanding build: a complete single-file Kanban app (~3,373 lines, no backend, localStorage persistence), assembled by the model over five steered phases, each a fresh session that read a spec + the current file and extended it — Playwright-verified between phases.

What works (all Playwright-driven against real interactions): boards / lists / cards CRUD, drag-and-drop cross-list moves with persistence, a markdown card modal with live preview, labels / due dates / checklists, keyboard shortcuts, and live search/filter — below, typing drag narrows to the matching card in real time:

Analysis. Over five phases the model needed only two corrections, each caught by Playwright and fixed by the model: an undeclared dragMoveOffsetX (1-line) and a search UI that was built but not yet wired to the filter logic.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Architecture details

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Usage

▌ llama.cpp (recommended)

Validated inference config (single-stream, thinking on):

llama-server \
  -m Qwopus3.6-27B-Fusion-BF16.gguf \
  -c 98304 \                 # 96K context; native is 262K. BF16 weights are ~55GB — size VRAM/RAM accordingly
  --parallel 1 \             # SINGLE stream — validated single-stream
  -ngl 999 \                 # all layers on GPU
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --reasoning on --jinja \   # thinking mode + the baked-in chat template
  --host 0.0.0.0 --port 8001

Recommended sampling: temperature 0.85–1.0 top_p 0.9.

Temperature — run it hot. Best capability lands at temp 0.85–1.0 (the finetune's natural operating point, and consistent with the agentic/SWE results). Then the surprising part: the merge itself appears to have repaired much of the low-temperature instability of the parent finetunes — Fusion stays clean at temps where the bases loop and degrade (4/4 termination at both 0.2 and 0.9, 10/10 across 0.5–0.9). So, unlike its parents, you can run it cooler for more determinism if you want, without the looping. Exactly why merging improves temperature stability isn't fully understood, but it's consistent across every test we ran. High temperature still yields the best results. Thinking mode is on by default and is where the model is strongest.

▌ As a coding backend (Claude Code / OpenAI-compatible clients)

The llama-server endpoint is OpenAI-compatible and has driven multi-file app builds end-to-end. If you point Claude Code at it, use a proxy that does native Anthropic↔OpenAI tool translation (text-encoded tools stall the agent).

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Evaluation — full results

Benchmarks were run on the Q4_K_M quant (thinking-on unless noted); the BF16 weights in this repo are the full-precision source those results derive from and should meet or exceed them.

Capability: HumanEval 94.5% · MBPP 87.9% · GSM8K 95.0% · MATH-500 62% (64.7% subset) · IFEval-strict 82.8%.

Stability (the axis that separated this project's winners from losers):

Agentic (SWE-bench Verified, astropy slice, thinking-on): Fusion 7/15 (47%) > equal-weight DARE-TIES 5/15 > DUS-38B frankenmerge 3/15.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Limitations

• Not yet rigorously evaluated. No large-scale safety alignment / RLHF / red-teaming. Inherits biases and limits of the Qwen base and both parent finetunes.
• The astropy +2 (7/15 vs 5/15) is within noise on n=15 — the geometry approach is validated as sound and directionally better, not a large agentic jump.
• On the hardest agentic instances the model can over-deliberate and hit the generation cap without submitting; capping max_tokens (~8K) mitigates this.
• Validated primarily at Q4_K_M; other quants should work but aren't separately benchmarked.

Found a problem? → please reach out on X: @kylehessling1.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Method credit & acknowledgments

The path to this model was not a straight line, and the credit is honest about that:

• Jackrong — base-model builder, collaborator, and good friend. This fusion stands entirely on his work: he built both parent finetunes (Qwopus3.6-27B-v2 and Qwopus3.6-27B-Coder) that this merge fuses. Beyond the models, his counsel and contributions throughout were priceless — this project simply does not exist without him. Go follow his work → huggingface.co/Jackrong.
• The initial naive approaches — a 38B depth-upscaled (DUS) frankenmerge and an equal-weight DARE-TIES merge from the shared base — were proposed by Fable. Both underdelivered. In hindsight, those plans conspicuously omitted the one nuance that turned out to be decisive: the lineage relationship between the parents (that the coder is a descendant of V2). Without that nuance, an equal-weight merge is essentially guaranteed to double-count V2's delta and import the destabilizing early-layer coding delta at full strength — which is exactly what went wrong.
• Grok 4.5 was instrumental in the final, working merge. Its analysis surfaced the parent→descendant lineage from the weight geometry and motivated the layer-weighted delta-scale recipe (protect early reasoning, inject coding late) that this model is built on.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

◆ Lineage / provenance

Qwen/Qwen3.6-27B  (shared base)
├── Qwopus3.6-27B-v2      (reasoning finetune)  ──┐
└── Qwopus3.6-27B-Coder   (code finetune,        │  ← descends from V2 (proj→1.0 in late layers)
     itself downstream of V2)                    │
                                                  ▼
        Qwopus3.6-27B-Fusion = V2 + α(L)·(Coder − V2),  α: 0.12 → 0.48 by depth

◆ License

A merge of derivatives of Qwen/Qwen3.6-27B; usage is governed by the Qwen base model license. The parent finetunes are personal research models. No third-party training data is redistributed here — only merged weights and the merge script. Comply with the upstream Qwen license.

◆ Reproduction

python merge_layerweighted.py --out ./Qwopus3.6-27B-Fusion --a-early 0.12 --a-late 0.48
# then convert + quantize with llama.cpp: convert_hf_to_gguf.py  →  llama-quantize ... Q4_K_M

If you build on the method (geometry-grounded, layer-weighted task-vector merging of a parent/descendant finetune pair), a mention of "FusionThink layer-weighted delta-scale merge" is appreciated.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

<div align="center"> <i>Built and evaluated locally on a single RTX 5090. Model produced by Kyle Hessling (<a href="https://x.com/kylehessling1">@kylehessling1</a>).</i> </div>