Qwen3-30B-A3B-YOYO-V5

The 5th-Generation Merged Model of YOYO-AI and the brand-new merging algorithm "yoyo_fusion" have been officially released!

Model Highlights:

• merge method: yoyo_fusion

• precision: dtype: bfloat16

• Context length: 262,144&1010000

Parameter Settings:

[!TIP] Temperature=0.7, TopP=0.8, TopK=20,MinP=0.

GitHub Repository:

YOYO-Fusion

Configuration:

The following configuration was used to produce this model:

from yoyo_fusion import run_merge

run_merge(
    model_paths=[
        "Qwen/Qwen3-30B-A3B-Instruct-2507",
        "Qwen/Qwen3-30B-A3B-Thinking-2507",
        "Qwen/Qwen3-Coder-30B-A3B-Instruct"
    ],
    output_dir="YOYO-AI/Qwen3-30B-A3B-YOYO-V5",
    anchor_index=0,
    config_dir=1,
    use_k_minus_one_truncation=True,
    use_geometric_median=True,
)

YOYO-Fusion: Robust Merging in Residual Subspace

Input

Given K≥2 weight tensors from models with identical architecture:
$$ {T^{(1)}, T^{(2)}, \dots, T^{(K)}}, \quad T^{(k)} \in \mathbb{R}^{d_1 \times \cdots \times d_n}, $$

---

Step 1: Flatten and RMS-normalize each tensor

Flatten each tensor into a vector and normalize by its RMS:
$$ x^{(k)} = \operatorname{flatten}(T^{(k)}) \in \mathbb{R}^D, \quad D = \prod_{i=1}^n d_i $$
$$ r_k = \operatorname{RMS}(x^{(k)}) = \sqrt{ \frac{1}{D} \sum_{i=1}^D (x^{(k)}_i)^2 + \varepsilon } $$
$$ u^{(k)} = \frac{x^{(k)}}{r_k + \varepsilon} $$

---

Step 2: Determine Center Point

Case A: Anchor Mode

$$ \mathbf{m} = \mathbf{u}_n $$

Case B: No Anchor Mode

• Subcase B1:

  Compute the geometric median via the Weiszfeld algorithm:

$$ \mathbf{m} = \arg\min_{\mathbf{y}} \sum_{i=1}^K | \mathbf{u}_i - \mathbf{y} |_2 $$

• Subcase B2:

  Use coordinate-wise median:

$$ m_j = \text{median}(u_{1,j}, u_{2,j}, \dots, u_{K,j}), \quad \forall j=1,\dots,D $$

---

Step 3: Compute residual matrix

$$ \mathbf{R} = \mathbf{U} - \mathbf{1}_K \mathbf{m}^\top \in \mathbb{R}^{K \times D} $$

---

Step 4: Early exit if residuals are negligible

If
$$ \max_k |R_{k,:}|_2 < 10^{-7}, $$
then set
$$ \mathbf{y}' = \mathbf{m} $$
and skip to Step 8. Otherwise, proceed.

---

Step 5: Perform SVD on residuals

Compute the thin SVD of R^⊤∈R^D×K:
$$ R^\top = U \Sigma V^\top $$
Let min(K−1,rank(R)), and take the first r' columns of U :
$$ U_{r'} = U[:, :r'] \in \mathbb{R}^{D \times r'} $$

---

Step 6: Compute energy-based scaling factor

Total energy:
$$ E_{\text{total}} = \sum_{i=1}^{\operatorname{rank}} \sigma_i^2 $$
Retained energy:
$$ E_{\text{retained}} = \sum_{i=1}^{r'} \sigma_i^2 $$
Energy ratio:
$$ p = \frac{E_{\text{retained}}}{E_{\text{total}} + \varepsilon} $$
Scaling factor (clamped for stability):
$$ \lambda = \min\left( \frac{1}{p + \varepsilon},\ 10.0 \right) $$

---

Step 7: Robust weighted averaging in subspace

Project residuals into subspace

$$ Z = R U_{r'} \in \mathbb{R}^{K \times r'} $$

Estimate robust scales

Per-coordinate MAD scale:
$$ s_j = 1.4826 \cdot \operatorname{median}{k} \left( |Z{k,j}| \right), \quad j = 1, \dots, r' $$
Per-model residual norm:
$$ |z_k| = |Z_{k,:}|2 $$
Global MAD scale:
$$ s{\text{global}} = 1.4826 \cdot \operatorname{median}_{k} \left( |z_k| \right) $$

Compute Tukey bisquare weights（c = 4.685）

Coordinate-wise weights:
$$ w^{\text{coord}}{k,j} = \left[ \max\left( 0,\ 1 - \left( \frac{|Z{k,j}|}{c \cdot s_j + \varepsilon} \right)^2 \right) \right]^2 $$
Global (per-model) weights:
$$ w^{\text{global}}k = \left[ \max\left( 0,\ 1 - \left( \frac{|z_k|}{c \cdot s{\text{global}} + \varepsilon} \right)^2 \right) \right]^2 $$
Combined weights:
$$ W_{k,j} = w^{\text{coord}}_{k,j} \cdot w^{\text{global}}_k $$

Compute robust consensus in subspace

$$ z^j = \frac{ \sum{k=1}^K W_{k,j} Z_{k,j} }{ \sum_{k=1}^K W_{k,j} + \varepsilon }, \quad j = 1, \dots, r' $$
Reconstruct robust residual:
$$ r^ = \lambda \cdot U_{r'} z^* \in \mathbb{R}^D $$
Final estimate in normalized space:
$$ y' = m + r^* $$

---

Step 8: Restore average RMS scale

Compute mean RMS across inputs:
$$ \bar{r} = \frac{1}{K} \sum_{k=1}^K r_k $$
Scale back:
$$ y = y' \cdot \bar{r} $$

---

Step 9: Final L2 norm alignment

Compute average L2 norm of original flattened tensors:
$$ \bar{n} = \frac{1}{K} \sum_{k=1}^K |x^{(k)}|_2 $$
Compute current norm:
$$ n_y = |y|_2 $$
Final scaling factor:
$$ \alpha = \frac{\bar{n}}{n_y + \varepsilon} $$
Scaled output vector:
$$ \hat{x} = \alpha \cdot y $$
Reshape to original tensor shape:
$$ \hat{T} = \operatorname{reshape}(\hat{x},\ (d_1, \dots, d_n)) $$