butterfly-tipping-point-51B

<p align="center"> <img src="scrapegoat.png" alt="ScrapeGoat" width="400"/> </p> <h1 align="center">Butterfly Tipping Point 51B</h1> <p align="center"> <em>A dual-track parallel transformer with learned inter-track gating</em> </p> <p align="center"> <a href="https://huggingface.co/scrapegoat/butterfly-tipping-point-51B"> <img src="https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow" alt="HuggingFace"> </a> <a href="https://github.com/scrapegoat/butterfly-tipping-point"> <img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"> </a> <img src="https://img.shields.io/badge/Params-50B-blue" alt="Parameters"> <img src="https://img.shields.io/badge/Layers-64-green" alt="Layers"> <img src="https://img.shields.io/badge/Context-262K-orange" alt="Context"> </p>

---

Overview

Butterfly Tipping Point (BTP) is a dual-track parallel transformer that runs two complete attention/MLP streams simultaneously at every layer and blends their outputs with a learned per-token gate — inspired by the corpus callosum connecting the brain's hemispheres.

Unlike traditional transformers where every layer applies the same attention mechanism, BTP maintains two parallel computation tracks:

A learned corpus callosum gate (attn_track_gate) produces a 2-logit sigmoid that determines how much each track contributes at every position, allowing the model to dynamically allocate computation.

Architecture

                              |
                              v
              +-------------------------------+
              |           EMBEDDING           |
              |   Vocab: 248,320              |
              |   Dimension: 5,120            |
              +---------------+---------------+
                              |
                              v
        +---------------------------------------------+
        |             DUALTRACK LAYER x64             |
        |                                             |
        |   +----------------+    +----------------+  |
        |   |    TRACK A     |    |    TRACK B     |  |
        |   |                |    |                |  |
        |   |      GDN       |    |      GQA       |  |
        |   |    (linear)    |    |     (full)     |  |
        |   +-------+--------+    +--------+-------+  |
        |           |                      |          |
        |           +----------+-----------+          |
        |                      |                      |
        |                 GATE [2]                    |
        |                      |                      |
        |                      v                      |
        |          +-------------------------+        |
        |          |     DUAL MLP BLEND      |        |
        |          |                         |        |
        |          | gA * MLP_A + gB * MLP_B |        |
        |          +------------+------------+        |
        |                       |                     |
        +-----------------------+---------------------+
                                |
                                v
                  +---------------------------+
                  |   RMSNorm -> LM Head      |
                  +-------------+-------------+
                                |
                                v
                            OUTPUT

Layer Layout

The 64 layers use a 3:1 alternation pattern (matching Qwen3.5's native design):

Every layer runs both tracks in parallel and blends via the learned gate. Full-attention layers handle the heavy lifting of global reasoning; linear-attention layers provide efficient streaming and local context.

Key Specifications

Track A: Gated Delta Net (Linear Attention)

Based on Qwen3.5's GatedDeltaNet, Track A implements a linear-complexity recurrent attention mechanism:

• 16 key heads × 128-dim (vs. Track B's 4 KV heads × 256-dim)
• 48 value heads × 128-dim (richer value representation)
• Conv1d kernel (dim=4) for local context mixing
• 1D state-space recurrence — maintains a fixed-size hidden state per head, enabling O(1) per-token inference
• Processes the full sequence in a single pass (no KV cache needed)

Track B: Full GQA Self-Attention

Standard Grouped Query Attention with:

• 24 query heads × 256-dim
• 4 KV heads × 256-dim (6:1 query/KV ratio)
• QK-Norm via GemmaRMSNorm for training stability
• Sigmoid output gate (attn_output_gate=true)
• Full causal masking over the context window
• KV cache for autoregressive generation

Corpus Callosum Gate

The inter-track gate is a learned linear layer producing 2 logits per token:

gate = sigmoid(hidden_states @ attn_track_gate.weight.T + attn_track_gate.bias)
# gate ∈ [0, 1]² per token — per-track blending weights

Initialization: Quantile-Balanced — re-centers gate weights to zero mean and L2-normalizes them so logits are content-driven O(1) rather than saturated. This prevents the gate from collapsing to a hard 0/1 routing (all-A or all-B) early in training.

MLP blending: Both tracks have their own MLP (Qwen2MoEMLP with gated SiLU). The final MLP output is:

mlp_output = gate[:, 0] * MLP_A(hidden) + gate[:, 1] * MLP_B(hidden)

Serving With SGLang

python3 -m sglang.launch_server \
  --model-path scrapegoat/butterfly-tipping-point-51B \
  --served-model-name butterfly_tipping_point \
  --host 0.0.0.0 --port 30000 \
  --attention-backend triton \
  --quantization fp8 \
  --mem-fraction-static 0.95 \
  --trust-remote-code

Environment Variables

API Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="butterfly_tipping_point",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the butterfly effect in chaos theory."}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)

Benchmark Results

<p align="center"> <img src="butterfly_tipping_point_50b_comparison.png" alt="Benchmark Comparison" width="700"/> </p>

How It Works

The Dual-Track Hypothesis

Traditional transformers process every token through the same mechanism at every layer. This is computationally expensive (quadratic attention) and doesn't allow the model to specialize different layers for different roles.

BTP addresses this by maintaining two parallel computation streams at every layer:

1. Track A (Butterfly Effect) — Fast, linear-complexity attention that maintains a fixed-size state. Like small perturbations in a chaotic system, it captures the "butterfly effect" of local token interactions.

2. Track B (Tipping Point) — Full quadratic attention that provides precise global reasoning. Like a phase transition at the "tipping point," it enables the model to synthesize information across the entire context.

The learned gate dynamically routes each token to the appropriate track based on content — simple tokens might get 90% Track A (fast), while complex reasoning tokens might get 50/50 or even 70% Track B.

Quantile-Balanced Gate Initialization

Without careful initialization, the gate quickly collapses to hard routing (always A or always B), defeating the purpose of dual tracks. The Quantile-Balanced initialization:

1. Re-centers each gate weight row to zero mean
2. L2-normalizes to unit norm
3. Scales by BTP_GATE_SCALE

This keeps both tracks active (~35-65%) and ensures the gate is driven by input content rather than weight magnitude.

Citation

@article{butterfly-tipping-point,
  title={Butterfly Tipping Point: Dual-Track Parallel Transformers with Learned Inter-Track Gating},
  author={ScrapeGoat Research},
  year={2026}
}

Coding Agent Framework

The most powerful AI coding agent on planet Earth is now a self-learning model agnostic harness.

/fast mode and more advanced skills enabled only with scrapegoat models

/graph - indexes your repo locally and auto-injects repo context. Excoder is always code aware.

/Agent-WebBridge - Browser automation via Agent-WebBridge-skill and Agent-WebBridge for QA automation and Browser Autonomy.

/dispatching-parallel-agents - To dispatch Swarm of agents for parallel work.

Team | Enterprise: run excoder, pick a username on first launch, then use /message @teammate to chat with org members in real time with secure encrypted messaging.

/redteam-skill & /pentest-skill for offensive and advanced cyber security capabilities.

• Application Security Testing — Detect and validate critical vulnerabilities in your applications
• Rapid Penetration Testing — Get penetration tests done in hours, not weeks, with compliance reports
• Bug Bounty Automation — Automate bug bounty research and generate PoCs for faster reporting
• CI/CD Integration — Run tests in CI/CD to block vulnerabilities before reaching production
• 🕸️ Web apps | Black-box, external-attacker recon → exploit (XBEN suite) | ✅
• 🚩 CTF | Hint-free, sandbox-jailed solves (Cybench) | ✅
• 🤖 Robotics / OT / embedded | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅
• 📂 Source code | White-box repo analysis with blind master-builder decomposition
• 💰 Smart contracts | Damn Vulnerable DeFi | ⚠️ reproduction
• ☁️ Cloud (IaC) | Misconfig-detection benchmark (cloud:bench) + opt-in cloud arsenal | 🚧 IaC-misconfig scaffolding — live-cloud exploitation
• 📱 Mobile | Built-in static analyzer (manifest misconfig + secret/cleartext detection, mobile:bench) + opt-in arsenal (mobsfscan/objection/drozer; frida gated)
• 🔩 Binary / RE | Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, binary:bench)

License

Apache 2.0

Acknowledgments

Built on top of SGLang serving framework and Qwen3.5 architecture. The GatedDeltaNet component is based on Yang et al., 2025.

---

<div align="center"> <strong>Left brain • Right brain • One mind. 🐐</strong> </div>