Qwen35b-agent-R2O3

<p align="center"> <img src="https://img.shields.io/badge/license-AGPL--3.0-red"> <img src="https://img.shields.io/badge/Qwen3.5-35B%20A3B-blue"> <img src="https://img.shields.io/badge/MoE-256%20experts-orange"> <img src="https://img.shields.io/badge/SVD-LoRA-ff69b4"> <img src="https://img.shields.io/badge/Ornith-0.3-green"> <img src="https://img.shields.io/badge/Agent-R2O3-black"> </p> <p align="center"><b>🚀 Qwen35b-Agent-R2O3 — Agent-R2 + Ornith (α=0.3)</b></p> <p align="center"><i>Built on Qwen35B-Agent-R2 with SVD-extracted Ornith LoRA. The best of both worlds.</i></p>

---

🧬 How This Model Was Built

┌─────────────────────────────────────────────────────────────┐
│              Qwen35b-Agent-R2O3 Construction                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Qwen35B-Agent-R2 (Base)  ────────── 70% weights kept       │
│       │                                                     │
│       ├── 7 LoRAs already fused:                            │
│       │   Opus | Fable | Routing | Tool | Math | Mythos     │
│       │   | ToolFmt (all trained via SFT)                   │
│       │                                                     │
│       └── + Ornith LoRA (α=0.3) ← SVD Weight-Diff          │
│                                                             │
│  Ornith-1.0-35B                         Qwen-AgentWorld     │
│       │                                       │             │
│       └──────── Weight-Diff SVD ──────────────┘             │
│                    │                                        │
│             ┌──────┴──────┐                                 │
│             │  LoRA r=32  │ → Merged at α=0.3               │
│             │  422 tensors│                                  │
│             └─────────────┘                                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Step 1: SVD Weight-Diff Extraction

We extract Ornith's unique knowledge by computing the weight difference between Ornith-1.0-35B and the shared Huihui-Qwen-AgentWorld base:

delta = W_ornith - W_base         # What Ornith learned
U, S, Vh = torch.linalg.svd(delta) # Decompose
lora_A = diag(S[:32].sqrt()) @ Vh[:32, :]
lora_B = U[:, :32] @ diag(S[:32].sqrt())

422 tensors extracted across:

• MLP layers (gate_proj, up_proj, down_proj) — knowledge execution
• Expert weights (256 MoE experts) — specialized routing (3D tensor → flatten → SVD → reshape)
• Shared expert — common computation
• Norms + lm_head — output calibration
• Attention layers skipped — R2 uses linear_attn vs Ornith's self_attn (incompatible architecture)

Step 2: Multi-LoRA Fusion

The extracted Ornith LoRA (r=32, α=64) is merged into Qwen35B-Agent-R2 at scale α=0.3:

merged = R2 * 0.7 + Ornith_LoRA * 0.3

This preserves 70% of R2's original capabilities (its 7 LoRAs) while adding 30% of Ornith's algorithm/reasoning strength.

Why α=0.3?

---

🔬 Technique: SVD Weight-Diff for MoE

MoE models (256 experts) require special handling for SVD extraction:

Expert tensor handling:

delta = W_a - W_b                          # [512, 2048, 256]
delta_flat = delta.transpose(0,2).reshape(-1, delta.shape[1])  # [131072, 2048]
U, S, Vh = torch.linalg.svd(delta_flat)
lora_B = U[:, :32] @ diag(S[:32].sqrt())   # [131072, 32]

# On merge: reconstruct
delta = lora_B @ lora_A                    # [131072, 2048]
delta = delta.reshape(512, 256, 2048).permute(0, 2, 1)  # [512, 2048, 256]

---

📊 What You Get

---

🏆 Why Agent-R2O3?

---

🚀 Usage

# llama.cpp
./llama-cli -m Qwen35b-agent-R2O3.Q4_K_M.gguf \
  -p "Hello" -n 100 --temp 0.6

# Full server with tool calling
./llama-server \
  -m Qwen35b-agent-R2O3.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8081 -c 262144 -ngl 99 \
  --cache-type-k bf16 --cache-type-v bf16 \
  --flash-attn on --tools all --cont-batching \
  --temp 0.6 --top-k 40 --top-p 0.9 \
  --min-p 0.05 --repeat-penalty 1.03 \
  --jinja

---

📦 Downloads

---

🙏 Acknowledgements

---

💖 Support

<p align="center"> <img src="https://huggingface.co/hotdogs/Qwen35B-Agent-R2/raw/main/donate.webp" alt="Bitcoin QR" width="256"> </p>

bc1qf27cyk3vmugcdyv9xdtuv5jwz37863crpj5c9v

---

Built with ❤️ by UKA — 18-year-old coder & cybersecurity expert