Our1-2b

Our1-2b (OLMo-2B + Engram + AttnRes)

Our1-2b is a deep & narrow ~2.65B parameter language model ($d_{\text{model}} = 2048$, 41 layers) trained on an Apache 2.0 clean-room foundation using top-$k$ distillation from allenai/Olmo-3.1-32B-Think.

Model Specifications

• Architecture: 41 Transformer layers (10 blocks $\times$ [3 SWA + 1 Global] + 1 Gather layer)
• Width & Heads: $d_{\text{model}} = 2048$, $d_{\text{ff}} = 7168$, 16 Query heads, 4 KV heads ($d_{\text{head}} = 128$, 4:1 GQA)
• Positional Encoding: Partial RoPE (p-RoPE @ 0.25), multi-$\theta$ (1M for blocks 1–5; 5M for blocks 6–10 and Gather)
• Attention Residuals (AttnRes): Globals-only (11 connection points across the network)
• Engram Sidecar: 16.98M host-resident rows ($N \in {1, 2, 3}$) with context-aware kv_sigmoid readout at Layer 3 output
• Training Runway: 104.86M tokens across 3,200 steps at $T = 8192, B = 4$ (32,768 tokens/step)
• Final Loss: 3.0734 (EMA 2.9556, instantaneous minimum 1.9193)

Loading the Model

Because of the custom AttnRes and Engram sidecar topology, load using the clean-room harness:

from train.src.config import load_model_config
from train.src.model.refit import RefitModel
from train.src.tools.base_ckpt import load_base_checkpoint

cfg = load_model_config('config.json')
model = RefitModel(cfg)
load_base_checkpoint(model, '.')