Fuse-2-MoE-GGUF

Fuse-2 MoE GGUF (Quantized)

⚠️ CRITICAL: These GGUF files do NOT work with stock llama.cpp. You MUST use a patched build. See Installation below.

Model Details

This is the full 12B Fuse4 MoE model (Qwen3.5-4B host + 192 pruned Qwen3.8-27B experts) converted to GGUF format for llama.cpp.

BF16 (unquantized, 17.8 GB) is available at Akahsizrr/Fuse-2-MoE-BF16.

Architecture

Fuse4 is built on Qwen3.5-4B as the host model, augmented with 192 SwiGLU experts pruned from Qwen3.8-27B:

• 32 layers total (layers 3–26 are MoE-augmented, the rest are dense)
• 8 experts per augmented layer, top-2 routing
• Host FFN acts as a shared expert
• Bridge projections (2560→5120 and 5120→2560) connect host and expert spaces
• RMSNorm applied to expert output before residual addition
• Expert contribution is gated and scaled (static scale ≈ 0.0184)

The conversion folds bridge_in into expert gate/up weights and bridge_out into expert down weights. The coding_norm (RMSNorm) is stored as a custom ffn_moe_norm tensor. A patched llama.cpp applies this RMSNorm and the static scale after the MoE computation.

Installation

Step 1: Clone llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

Step 2: Apply the Fuse4 patches

You need to patch 5 files in llama.cpp to add FFN_MOE_NORM tensor support and the build_layer_ffn_fuse4 graph function.

Patch 1 — src/llama-arch.h

Add LLM_TENSOR_FFN_MOE_NORM to the llm_tensor enum (after FFN_DOWN_SHEXP):

    LLM_TENSOR_FFN_DOWN_SHEXP,
    LLM_TENSOR_FFN_MOE_NORM,

Patch 2 — src/llama-arch.cpp

Add the tensor name mapping (after ffn_down_shexp):

    { LLM_TENSOR_FFN_DOWN_SHEXP,                         "blk.%d.ffn_down_shexp" },
    { LLM_TENSOR_FFN_MOE_NORM,                          "blk.%d.ffn_moe_norm" },

Add the tensor info entry (after FFN_DOWN_SHEXP info):

    {LLM_TENSOR_FFN_DOWN_SHEXP,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL_MAT}},
    {LLM_TENSOR_FFN_MOE_NORM,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},

Patch 3 — src/llama-model.h

Add the ffn_moe_norm field to struct llama_layer (after ffn_up_shexp):

    struct ggml_tensor * ffn_up_shexp       = nullptr;
    struct ggml_tensor * ffn_moe_norm       = nullptr;  // Fuse4: RMSNorm after MoE

Add the build_layer_ffn_fuse4 declaration to the llama_model_qwen35moe::graph class in src/models/models.h:

        ggml_tensor * build_layer_ffn(
                    ggml_tensor * cur,
                            int   il);

        ggml_tensor * build_layer_ffn_fuse4(
                    ggml_tensor * cur,
                    ggml_tensor * ffn_residual,
                            int   il);

Patch 4 — src/models/qwen35moe.cpp

4a. Load the tensor in load_block_trunk (after ffn_down_shexp):

        layer.ffn_down_shexp     = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP,     "weight", il), { n_ff_shexp, n_embd }, flags);
        layer.ffn_moe_norm       = create_tensor(tn(LLM_TENSOR_FFN_MOE_NORM,        "weight", il), { n_embd }, TENSOR_NOT_REQUIRED);

4b. In the graph build function (around line 220), replace the build_layer_ffn call with build_layer_ffn_fuse4:

        // MOE FFN layer
        // Fuse4: pass ffn_residual so build_layer_ffn_fuse4 can compute host_hidden = ffn_residual + shared_expert
        cur = build_layer_ffn_fuse4(attn_post_norm, ffn_residual, il);
        cb(cur, "ffn_out", il);

4c. Add the build_layer_ffn_fuse4 function before build_layer_ffn:

ggml_tensor * llama_model_qwen35moe::graph::build_layer_ffn_fuse4(ggml_tensor * cur, ggml_tensor * ffn_residual, const int il) {
    GGML_ASSERT(model.layers[il].ffn_gate_inp != nullptr);

    if (model.layers[il].ffn_up_shexp != nullptr && model.layers[il].ffn_moe_norm != nullptr) {
        // 1. Compute shared expert (host FFN) — no gating, always active
        ggml_tensor * ffn_shexp =
            build_ffn(cur,
                model.layers[il].ffn_up_shexp, NULL, model.layers[il].ffn_up_shexp_s,
                model.layers[il].ffn_gate_shexp, NULL, model.layers[il].ffn_gate_shexp_s,
                model.layers[il].ffn_down_shexp, NULL, model.layers[il].ffn_down_shexp_s,
                NULL, LLM_FFN_SILU, LLM_FFN_PAR, il);
        cb(ffn_shexp, "ffn_shexp", il);

        // 2. host_hidden = ffn_residual + shared_expert (what experts/router see in Fuse4)
        ggml_tensor * host_hidden = ggml_add(ctx0, ffn_residual, ffn_shexp);
        cb(host_hidden, "host_hidden", il);

        // 3. MoE experts operate on host_hidden
        ggml_tensor * moe_out =
            build_moe_ffn(host_hidden,
                model.layers[il].ffn_gate_inp,
                model.layers[il].ffn_up_exps,
                model.layers[il].ffn_gate_exps,
                model.layers[il].ffn_down_exps,
                nullptr, n_expert, n_expert_used,
                LLM_FFN_SILU, true,
                hparams.expert_weights_scale,
                LLAMA_EXPERT_GATING_FUNC_TYPE_SOFTMAX, il,
                nullptr, model.layers[il].ffn_gate_up_exps,
                model.layers[il].ffn_up_exps_s,
                model.layers[il].ffn_gate_exps_s,
                model.layers[il].ffn_down_exps_s);
        cb(moe_out, "ffn_moe_out", il);

        // 4. RMSNorm on MoE output
        moe_out = build_norm(moe_out, model.layers[il].ffn_moe_norm, nullptr, LLM_NORM_RMS, il);
        cb(moe_out, "ffn_moe_normed", il);

        // 5. Apply static scale (gate * scale ≈ 0.0184)
        moe_out = ggml_scale(ctx0, moe_out, 0.018421f);
        cb(moe_out, "ffn_moe_scaled", il);

        // 6. Return shared_expert + moe_expert (residual added by caller)
        cur = ggml_add(ctx0, ffn_shexp, moe_out);
        cb(cur, "ffn_out", il);
    } else {
        // Standard path for non-Fuse4 layers
        cur = build_layer_ffn(cur, il);
    }
    return cur;
}

Patch 5 — gguf-py/gguf/constants.py

Add FFN_MOE_NORM to the MODEL_TENSOR enum:

    FFN_DOWN_SHEXP       = auto()
    FFN_MOE_NORM         = auto()

Add the tensor name mapping:

    MODEL_TENSOR.FFN_DOWN_SHEXP:            "blk.{bid}.ffn_down_shexp",
    MODEL_TENSOR.FFN_MOE_NORM:              "blk.{bid}.ffn_moe_norm",

Add MODEL_TENSOR.FFN_MOE_NORM to the MODEL_ARCH.QWEN35MOE tensor list.

Patch 6 — gguf-py/gguf/tensor_mapping.py

Add the HF→GGUF tensor name mapping:

        MODEL_TENSOR.FFN_MOE_NORM: (
            "model.layers.{bid}.mlp.moe_norm",                        # fuse4
        ),

Step 3: Build llama.cpp

# CPU only
cmake -B build -DGGML_CUDA=OFF
cmake --build build --config Release -j --target llama-cli

# With CUDA (requires CUDA Toolkit 12.6+)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80
cmake --build build --config Release -j --target llama-cli

Step 4: Download a GGUF and run

# Download (from this repo)
huggingface-cli download Akahsizrr/Fuse-2-MoE-GGUF Fuse-2-MoE-Q4_K_M.gguf --local-dir ./models

# Run on GPU
./build/bin/llama-cli \
  -m ./models/Fuse-2-MoE-Q4_K_M.gguf \
  -c 4096 -n 256 \
  -ngl 99 \
  -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n"

Limitations

• Base model only: This GGUF was converted from the base fuse-2-boosted checkpoint, not the fine-tuned version (which was lost). Output quality may be lower than the fine-tuned model.
• Router gating: The original Fuse4 model uses sqrtsoftplus routing with a -2.0 bias and weight normalization. The GGUF uses softmax routing instead, which is an approximation.
• Residual clamping: The original model clamps the expert delta to 10% of the residual norm. This is not implemented in the GGUF (the static scale handles magnitude control instead).
• Expert output normalization: The original model normalizes expert outputs to match input scale. This is not replicated (RMSNorm handles normalization).
• Requires patched llama.cpp: These GGUFs will fail to load or produce garbage on unpatched llama.cpp.

License

Apache 2.0 (inherited from the base Qwen models and Fuse4 architecture).