qwen3.5-9b-triple-loop-fase2

Qwen3.5-9B Triple-Loop (Phase 2 — RL Step 30)

An advanced reasoning and agentic model based on Qwen/Qwen3.5-9B featuring LoopSplit (Triple-Loop Recurrence) and fine-tuned with Reinforcement Learning (GRPO / R1-Zero).

This checkpoint marks Phase 2, which introduces 30 steps of targeted Reinforcement Learning on an NVIDIA H100 SXM5 80GB GPU, sharpening tool-calling execution, structured chain-of-thought (<think>...</think>), and contrastive reasoning, while meticulously preserving the 64-layer virtual recurrent dynamics established in Phase 1.

---

🔬 Disambiguating the Triple-Loop Architecture

There is often confusion regarding how recurrent layer models operate at inference time. Here is the exact architectural specification:

Physical Storage vs. Logical Execution Graph

1. Physical Model (Storage & VRAM footprint):

   • Contains standard 32 physical transformer blocks (blk.0 to blk.31).
   • Physical parameter size is identical to stock Qwen3.5-9B (~9.2 billion parameters, ~17.9 GB in bf16, ~6.02 GB in Q5_K_M).
   • Fully loadable by standard libraries (Hugging Face Transformers, stock vLLM, default llama.cpp).

2. Logical Execution (The 64-Layer Virtual Loop):

   • Prefix (Blocks 0–7): Executed once (1×). Establishes syntactic parsing and high-level query embeddings.
   • Middle Loop (Blocks 8–23, 16 blocks): Executed three consecutive times (3×) with shared weights ($r_0 \to r_1 \to r_2$).
   • Suffix (Blocks 24–31): Executed once (1×). Projects representations toward vocabulary logits through the final RMSNorm.
   • Total Virtual Depth: $8 + (16 \times 3) + 8 = \mathbf{64\text{ logical layers}}$.

                     ┌──────────────────────────────────────────────┐
                     │           Middle Loop (Blocks 8..23)         │
                     │                                              │
                     │   ┌──────────┐   ┌──────────┐   ┌──────────┐ │
Prompt ──> [0..7] ──>│──>│ Pass 0   │──>│ Pass 1   │──>│ Pass 2   │─│──> [24..31] ──> Logits
 (Prefix, 1x)        │   │ (Hypo)   │   │ (Reflect)│   │ (Synth)  │ │     (Suffix, 1x)
                     │   └──────────┘   └──────────┘   └──────────┘ │
                     │         ▲              ▲              ▲      │
                     │         └──────────────┴──────────────┘      │
                     │             Shared Weights in Memory         │
                     └──────────────────────────────────────────────┘

Dual Operating Modes

• Mode A: Native / Distilled Pass (32 layers): Running the model without loop-aware runtimes passes tokens once through layers 0 to 31. Because the model was trained with Deep Supervision across loop stages, the 32-layer pass already yields strong baseline results.
• Mode B: True Triple-Loop Recurrence (64 logical layers): Enabled via the included engine/llamacpp_patch/ or native LoopSplit engines using --override-kv qwen35.triple_loop=int:1.
  • Memory Efficiency: Weights are aliased in memory via TENSOR_DUPLICATED (zero extra VRAM required for weights).
  • Context State: KV cache allocates 64 slots (one per logical depth), allowing each pass of the middle block to hold and transform distinct recurrent hidden states.

---

📊 Empirical Recurrent Dynamics: Inside the 3 Loops

We inspected the internal entropy and latent vector velocity ($\Delta h / h$) across an ambiguous distributed systems prompt:

---

⚡ Phase 2: Reinforcement Learning (Truth in Advertising)

We believe in complete transparency regarding training depth and parameters:

• What was done: Exactly 30 optimizer steps of Full Fine-Tuning Reinforcement Learning (GRPO / R1-Zero with group size $G=4$).
• Compute Hardware: 1× NVIDIA H100 SXM5 80GB GPU hosted on Modal.
• Optimizer & Hyperparameters: AdamW, learning rate $1.0 \times 10^{-6}$, gradient clipping 1.0, contrastive advantage scoring over verifiable multi-turn tool calling and code reasoning tasks.
• Why 30 steps?
  1. Behavioral Alignment: Phase 1 provided deep representation distillation (2,100 steps from Qwen3.8-27B). Phase 2 targeted output calibration: strict <tool_call> schema adherence, XML encapsulation, and CoT reasoning discipline.
  2. Preserving Recurrent Representations: Excessive RL steps with synthetic rewards often induce catastrophic forgetting of intermediate latent topologies. By limiting to 30 steps at $10^{-6}$ LR, the model achieved 65.6% benchmark success and 100% contrastive group discrimination without degrading loop dynamics.

Empirical Weight Drift (Phase 1 vs. Phase 2)

Layer-by-layer weight cosine similarity and relative Euclidean shift ($\Delta W / W$) confirm that representations were preserved:

• Prefix Blocks (0–7): Drift: 1.22% | Cosine Similarity: 0.99992
• Middle Recurrent Blocks (8–23): Drift: 1.17% | Cosine Similarity: 0.99993
• Suffix Blocks (24–31): Drift: 1.21% | Cosine Similarity: 0.99992
• Language Head (output.weight): Drift: 0.10% | Cosine Similarity: 0.9999995

Over 98.8% of the Phase 1 weight geometry remains intact, while tool-calling fidelity and reasoning stability are noticeably sharper.

---

📦 Files in this Repository

• model.safetensors — Full precision bfloat16 Hugging Face weights (~17.9 GB, 32 physical layers, mtp_num_hidden_layers: 0).
• config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja — Full tokenizer & config files.
• qwen35-9b-triple-loop-fase2-step30-Q5_K_M.gguf — Quantized GGUF (Q5_K_M, ~6.02 GB, exact 32-block count for seamless patch execution).
• engine/llamacpp_patch/ — Source patch and instructions to compile/run 64-layer Triple-Loop in llama.cpp.

---

🚀 How to Run

1. True 64-Layer Triple Loop with Patched llama.cpp (Recommended)

Apply the patch provided in engine/llamacpp_patch/ to llama.cpp (commit 53ed051ce) and build:

# Start server with 64 logical layers and full GPU offload:
llama-server -m qwen35-9b-triple-loop-fase2-step30-Q5_K_M.gguf \
  --override-kv qwen35.triple_loop=int:1 \
  -ngl 99 -c 32768 --port 8080 -fa 1 \
  --cache-type-k q8_0 --cache-type-v q8_0

Verify in server console:

qwen35: triple-loop enabled: 32 physical blocks -> 64 logical layers

2. Standard 32-Layer Inference (Stock llama.cpp or vLLM)

llama-server -m qwen35-9b-triple-loop-fase2-step30-Q5_K_M.gguf \
  -ngl 99 -c 32768 --port 8080

3. Recommended Sampling Parameters

• Temperature: 0.6
• Top-P: 0.95
• Top-K: 40
• Repeat Penalty: 1.08

---

🛠️ Tool Calling & Agentic Prompt Template

The model supports function calling via XML encapsulation:

<|im_start|>system
# Tools
You have access to the following functions:
<tools>
{"name": "bash", "description": "Execute bash commands", "parameters": {"properties": {"command": {"type": "string"}}, "required": ["command"], "type": "object"}}
</tools>
<|im_end|>
<|im_start|>user
List the files in the directory.<|im_end|>
<|im_start|>assistant
<think>
The user wants to inspect current directory contents. I should invoke the bash tool with 'ls -la'.
</think>
<tool_call>
<function=bash>
<parameter=command>
ls -la
</parameter>
</function>
</tool_call><|im_end|>

---

📜 Citation & Credits

• Base Architecture: Qwen/Qwen3.5-9B
• Teacher Model (Phase 1 Distillation): Qwen/Qwen3.8-27B
• Fine-Tuning & Architecture Research: Lordnyx