MiMo-Ornith-9B-AGSI

<div align="center">

</div>

---

📌 Executive Summary

MiMo-Ornith-9B-AGSI is a non-linear parameter-space synthesis of two leading fine-tuned models derived from the Qwen 9B architecture:

• XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B: A distillation checkpoint optimized for dense mathematical deduction, SWE-bench verified programmatic problem-solving, and long-horizon chain-of-thought (CoT) reasoning.
• ornith-ai/Ornith-1.5-9B: A reinforcement-learning-driven agentic model specializing in terminal/CLI mastery, autonomous bash execution, self-debugging loops, and GrandCode algorithmic generation.

Rather than relying on naive linear averaging (LERP) or global spherical interpolation (SLERP)—which flatten parameter tensors into uncalibrated vectors—this model was fused using Adaptive Geodesic Spectral Interpolation (AGSI). AGSI executes row-wise hyperspherical geodesics on decoupled directional manifolds, enforces second-order spectral energy conservation, shields against anti-phase gradient interference, and dynamically modulates parameters across depth via a $C^2$-continuous quintic smoothstep curve.

---

🔬 The Mathematics of AGSI

Standard model interpolation techniques often suffer from:

1. Frobenius Attenuation: Convex parameter averaging causes systematic shrinkage of matrix norms ($|(1-t)W_A + tW_B|_F < |W|_F$), resulting in signal degradation across 32 transformer layers.
2. Isotropic Collapsing: Flattening multi-head projections into a single 1D vector treats distinct semantic subspaces as an isotropic sphere, corrupting specialized attention head alignments.
3. Anti-Phase Annihilation: Conflicting updates between reinforcement learning (Ornith) and knowledge distillation (MiMo) cause destructive cancellation when vectors point in opposing directions ($\cos\theta < 0$).

AGSI resolves these issues through a five-stage manifold interpolation framework:

---

1. Direction-Magnitude (DoRA) Decoupling

Linear layers compute transformations $y = x W^T$, where individual rows $W_i \in \mathbb{R}^{D_{\text{in}}}$ represent the hyperplanes of specific neurons. AGSI isolates radial feature scale from directional orientation:

$$m^{(i)} = |W^{(i)}|2 = \sqrt{\sum{j=1}^{D_{\text{in}}} (W_{ij})^2}$$

$$u^{(i)} = \frac{W^{(i)}}{m^{(i)} + \epsilon}, \quad \text{where } u^{(i)} \in S^{D_{\text{in}}-1}$$

By constraining directional updates to the unit hypersphere $S^{D_{\text{in}}-1}$, the model preserves the angular separation of neuron receptive fields.

---

2. Row-Wise Hyperspherical Geodesics ($S^{D-1}$)

For each individual neuron row $i$, the angular geodesic distance between checkpoint trajectories is calculated directly in its tangent space:

$$\theta_i = \arccos\left(\text{clamp}\left(\langle u_A^{(i)}, u_B^{(i)} \rangle, -1 + \delta, 1 - \delta\right)\right)$$

Rather than using Euclidean displacement, the directional basis moves along the great-circle arc:

$$u_{\text{fused}}^{(i)} = c_A^{(i)} u_A^{(i)} + c_B^{(i)} u_B^{(i)}$$

where the geodesic velocity coefficients are defined as:

$$c_A^{(i)} = \frac{\sin((1 - t)\theta_i)}{\sin\theta_i + \epsilon}, \quad c_B^{(i)} = \frac{\sin(t \theta_i)}{\sin\theta_i + \epsilon}$$

If $\sin\theta_i \to 0$ (quasi-collinear vectors), the interpolation smoothly transitions to normalized linear blending:

$$\lim_{\theta \to 0} u_{\text{fused}}^{(i)} = (1 - t)u_A^{(i)} + t u_B^{(i)}$$

---

3. Anti-Phase Interference Shielding

When RL optimization (Ornith) and distillation gradients (MiMo) pull in opposing directions ($\theta_i > 90^\circ, \cos\theta_i < 0$), standard spherical combination results in substantial cancellation:

$$|u_A + u_B| = \sqrt{2 + 2\cos\theta} < \sqrt{2} \approx 1.414 \quad (\text{vs. } 2.0 \text{ when aligned})$$

AGSI monitors the directional dot product against a critical interference bound ($\tau = -0.05$, corresponding to $\theta \approx 92.86^\circ$). If destructive cancellation occurs:

$$\text{Conflict Condition: } \langle u_A^{(i)}, u_B^{(i)} \rangle < \tau$$

$$\gamma_A^{(i)} = \frac{m_A^{(i)}}{m_A^{(i)} + m_B^{(i)} + \epsilon}$$

$$u_{\text{fused}}^{(i)} = \begin{cases} 0.85, u_A^{(i)} + 0.15, u_B^{(i)}, & \text{if } \gamma_A^{(i)} \ge 0.5 \ 0.15, u_A^{(i)} + 0.85, u_B^{(i)}, & \text{if } \gamma_A^{(i)} < 0.5 \end{cases}$$

This dominance gate projects conflicting features toward the checkpoint exhibiting higher parameter variance, preventing the formation of dormant neurons.

---

4. Quadratic Spectral Energy Invariant

In deep autoregressive networks normalized by RMSNorm, layer output variance relies heavily on parameter norm preservation:

$$\mathbb{E}[|W x|2^2] \approx \frac{1}{D{\text{in}}} |W|_F^2 , \mathbb{E}[|x|_2^2]$$

To maintain stable forward-pass activations without logit saturation or signal decay, AGSI scales the interpolated direction vector by the root-mean-square energy:

$$m_{\text{target}}^{(i)} = \sqrt{(1 - t)(m_A^{(i)})^2 + t(m_B^{(i)})^2}$$

$$W_{\text{candidate}}^{(i)} = m_{\text{target}}^{(i)} \cdot \frac{u_{\text{fused}}^{(i)}}{|u_{\text{fused}}^{(i)}|_2}$$

Finally, a global Frobenius norm correction is applied across the full matrix:

$$W_{\text{final}} = W_{\text{candidate}} \times \left( \frac{\sqrt{(1 - t)|W_A|_F^2 + t|W_B|F^2}}{|W{\text{candidate}}|_F + \epsilon} \right)$$

---

5. Quintic Smoothstep Depth & Functional Block Routing

The mixing coefficient $t$ is non-static. It is governed by a $C^2$-continuous quintic polynomial function across the 32 transformer layers, augmented by functional block biases:

$$\xi = \frac{l}{L - 1} \in [0, 1], \quad l \in {0, 1, \dots, 31}$$

$$S(\xi) = 6\xi^5 - 15\xi^4 + 10\xi^3$$

$$t(l, \text{block}) = \text{clamp}\left(t_{\text{base}} + \Delta t \cdot \left(S(\xi) - 0.5\right) + \beta_{\text{block}}, , 0.0, , 1.0\right)$$

Layer Profile Breakdown:

• Layers 0–7 ($t \approx 0.41 - 0.44$) — Syntactic Anchoring: Biased toward MiMo-V2.6 to preserve input parsing, stable token embeddings, and baseline representations.
• Layers 8–23 ($t \approx 0.45 - 0.51$) — Cognitive & Algorithmic Core: Near-equilibrium blending. Mathematical deductions and algorithm planning synthesize with Ornith's tool-use reasoning.
• Layers 24–31 ($t \approx 0.52 - 0.55$) — Action Policy & Output Heads: Biased toward Ornith-1.5 to prioritize terminal command generation, tool-calling syntax, and execution policies.
• Attention vs. MLP Decoupling:
  • Attention Projections (q, k, v, o, linear_attn): $\beta_{\text{attn}} = +0.04$ (Ornith priority for context selection and environment state tracking).
  • MLP / Feed-Forward (gate, up, down_proj): $\beta_{\text{mlp}} = -0.04$ (MiMo priority for factual retention and associative coding memories).

---

⚙️ AGSI Configuration Profile

# The Golden SOTA Preset applied during parameter synthesis
BASE_ORNITH_RATIO        = 0.48   # Foundational balance (52% MiMo / 48% Ornith)
DEPTH_MODULATION         = 0.14   # Dynamic span across transformer depth
ATTN_ROUTING_BIAS        = 0.04   # Attention block offset (favors Ornith)
MLP_ROUTING_BIAS         = -0.04  # MLP block offset (favors MiMo)
ANTI_PHASE_THRESHOLD     = -0.05  # Critical angle cutoff (θ = 92.86°)
SPECTRAL_ENERGY_EXPONENT = 2.0    # Second-order L2 moment conservation

---

🚀 Deployment & Inference

This model is compatible with systems supporting the Qwen 9B architecture (e.g., standard Hugging Face transformers, vLLM, SGLang).

Serving with vLLM (Recommended)

To run the model with vLLM, including support for streaming reasoning tokens (<think>) and agentic tool calls:

vllm serve DEST_REPO_ID \
  --port 8000 \
  --model DEST_REPO_ID \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser qwen3 \
  --tool-call-parser hermes

Python Inference via Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DEST_REPO_ID"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {
        "role": "system",
        "content": "You are a master systems engineer and competitive programmer. Solve problems using precise step-by-step reasoning enclosed in <think> tags, then execute terminal commands or output production code."
    },
    {
        "role": "user",
        "content": "Write an optimized eBPF program in C that traces network socket latency outliers (>100ms) and provides a Python BCC analysis script."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    repetition_penalty=1.05
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

🎯 Prompt Formatting & Chat Template

This model uses the unified Qwen chat template, configured to separate internal reasoning steps from final outputs:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Write a bash script to monitor memory usage.<|im_end|>
<|im_start|>assistant
<think>
1. Identify target metrics: available vs. used memory.
2. Use /proc/meminfo or 'free -m' for portability.
3. Handle logging and alert thresholds.
</think>
Here is the monitoring script:

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD=85
CURRENT=$(free | awk '/Mem:/ {printf("%.0f"), $3/$2 * 100}')

if [ "$CURRENT" -gt "$THRESHOLD" ]; then
    echo "WARNING: Memory usage at ${CURRENT}%"
fi

<|im_end|>

---

⚖️ License & Attribution

• Base Model Checkpoints:
  • Checkpoint A: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
  • Checkpoint B: ornith-ai/Ornith-1.5-9B
• Underlying Architecture: Qwen Series (Alibaba Cloud)
• License: Apache 2.0