SiameseNorm-DepthAttention

OLMo 3 1B and 3B — SiameseNorm + DepthAttention and Matched Baselines

This repository mirrors the four-stage OLMo 3 training pipelines for both the SiameseNorm + DepthAttention architecture and matched pure OLMo 3 baselines. Each model size contains five checkpoints: Stage 1, Stage 2, Stage 3, Stage 4 Think SFT, and Stage 4 Instruct SFT.

The individual repositories are grouped in the SiameseNorm-DepthAttention collection.

OLMo 3 3B checkpoints

The same 3B artifacts are mirrored here under:

OLMo 3 1B checkpoints

The 1B artifacts retain their original layout:

Stage 3 and Stage 4 use the frozen 65,536-token configuration. YaRN applies to Full-Attention layers; SWA layers retain the original RoPE and a 4,096-token window.

Loading SiameseNorm + DepthAttention

The modified checkpoints require transformers>=4.57.6,<5 and trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ArchSpace-Collection/SiameseNorm-DepthAttention"
subfolder = "olmo3/3b/stage4/instruct"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

Loading the matched baseline

The baseline uses Transformers' official Olmo3ForCausalLM implementation and does not require remote code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ArchSpace-Collection/SiameseNorm-DepthAttention"
subfolder = "olmo3/3b/baseline/stage4/instruct"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder=subfolder,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    subfolder=subfolder,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

fix_mistral_regex=False is intentional and preserves the tokenizer behavior used during training.

Matched 3B Instruct SFT evaluation

On the final eight-task objective evaluation, SiameseNorm + DepthAttention improves the matched OLMo 3 3B baseline average from 48.23 to 51.57.

Preliminary matched 1B evaluation

The final eight-task 1B objective-evaluation averages are nearly tied: 36.9 for SiameseNorm + DepthAttention and 36.8 for the matched baseline.

Architectures

Both sizes use the [SWA, SWA, SWA, Full] attention pattern, a 4,096-token sliding window, and reordered RMSNorm. The modified variant additionally enables SiameseNorm and sparse cross-layer DepthAttention; the matched baseline disables both modifications.

These Hugging Face artifacts are intended for inference and generation. Exact continuation of the native distributed training objective should use the MindSpeed/Megatron training pipeline.