dragonling-gate-adapter

dragonling-gate-adapter

Gate adapter weights for chili-lab/Ouro-hybrid-1.4B, trained using the dragonling BDH multiplicative gating method.

This adapter replaces the additive FFN residual in a looped transformer with a learned sparse multiplicative gate:

# original (Ouro-hybrid / LT2)
h = h + feed_forward(ffn_norm(h))

# dragonling gate
h = h + relu(gate_proj(ffn_norm(h))) * feed_forward(ffn_norm(h))

The gate is initialised to identity (gate_proj.weight = 0, gate_proj.bias = 1), so at step 0 the patched model is bit-identical to the pretrained baseline.

---

Adapter contents

The adapter does not include the base model weights. Download chili-lab/Ouro-hybrid-1.4B separately.

---

Results summary

Five experiments across two domains. The gate consistently outperforms an lm_head-only baseline at identical parameter budget (100.66 M vs 100.71 M).

This adapter corresponds to Experiment 5 (5 000 steps, GSM8K answer-only loss, cosine LR decay 1e-3 → 5e-5). Val loss: 8.68 → 1.57 nats.

Note on accuracy: GSM8K exact-match accuracy is 0 % across all adapter configurations. The gate adapter learns the format and style of mathematical reasoning but does not restructure the attention mechanism needed to ground reasoning in the specific input question. See the paper for full discussion.

---

How to use

Requires bdh_lt2/ from the dragonling repo.

from bdh_lt2.student_model import register_student_model
from bdh_lt2.gating import apply_bdh_gating
from transformers import AutoModelForCausalLM, AutoTokenizer
from safetensors.torch import load_file

# 1. Register the custom model class and load the base model
register_student_model()
model = AutoModelForCausalLM.from_pretrained(
    "chili-lab/Ouro-hybrid-1.4B",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "chili-lab/Ouro-hybrid-1.4B", trust_remote_code=True
)

# 2. Inject gate architecture (identity init — bit-identical to base at this point)
apply_bdh_gating(model, init_identity=True)

# 3. Load trained gate weights from this adapter
gate_sd = load_file("adapter_weights.safetensors")
model.load_state_dict(gate_sd, strict=False)

# 4. Use the model
inputs = tokenizer("Q: What is 2 + 2?\nA:", return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=False,
    repetition_penalty=1.3,
    use_cache=False,   # GDN has no KV cache in the pure-PyTorch implementation
)
print(tokenizer.decode(out[0]))

---

Training details

---

License

This adapter is released for research use only, consistent with the license of chili-lab/Ouro-hybrid-1.4B. The adapter weights (gate projections only) are original trained parameters; no base model weights are included.

---

Paper

Carter, J. & Knox, S. (2026). Multiplicative Gate Adapters for Looped Transformers. Zenodo. https://zenodo.org/records/21311787

Citation

@misc{dragonling2026,
  title  = {Multiplicative Gate Adapters for Looped Transformers},
  author = {Carter, Jackie and Knox, Samuel},
  year   = {2026},
  url    = {https://zenodo.org/records/21311787}
}

---

Upstreams

• LT2 — chili-lab/LT2, arXiv 2605.20670
• BDH — pathwaycom/bdh, arXiv 2509.26507
• Ouro — ByteDance/Ouro-1.4B, arXiv 2510.25741