ARO-BabyLM-1B

ARO-BabyLM-1B: Asymmetric Relational Operator

The Ultimate Present

日本語概要

A 116.6M-parameter causal language model trained from scratch under the BabyLM 2026 Strict word-exposure budget.

ARO stands for Asymmetric Relational Operator. The name describes the architecture's central computation. Asymmetric means that forward and reverse relations are kept distinct rather than collapsed into one symmetric relation. Relational refers to the learned latent modes through which the model compares its evolving Present state with several internal evidence sources. Operator emphasizes that these relations do more than describe the state: they condition the feed-forward transformation and help reorient the next Present state.

The architecture maintains one evolving Present state where three kinds of evidence meet: exact causal retrieval from the visible context, a compressed recurrent memory of the past, and training-time pressure to preserve information useful for predicting several future horizons. Inference remains strictly causal and uses only the visible prefix.

ARO's conceptual origin lies in two sources: a near-death-experience account that described time as a single panorama centered on the observer's present, and the Buddhist principle of dependent origination, in which phenomena arise through chains of relation. These ideas were translated into testable computation: one evolving Present state, multiple forms of evidence arriving from the past, a second reading of the same past from an updated Present, twelve learned relation modes with forward and reverse structure, and training pressure directed toward several future horizons.

This repository is the BabyLM specialization of ARO. It contains the 1B-word-exposure checkpoint, a native 16,384-token byte-level BPE tokenizer, custom Transformers integration, and the architecture source required to reproduce inference.

Evaluation status — 1 September 2026: pretraining is complete and the final checkpoint has passed its SHA-256 integrity check. Official BabyLM 2026 full zero-shot and Reading evaluation is running. The table below reports the verified 500M intermediate checkpoint on the official fast checkpoint suite. Final-model results will replace the pending section after verification.

Highlights

• 116,647,315 parameters with tied input and output embeddings.
• 1,000,000,000 words of total training exposure, drawn from a fixed 100M-word corpus over ten passes.
• 1,513,881,600 subword tokens processed in 11,550 optimizer steps.
• A hybrid causal architecture combining local/global attention, GatedDeltaNet memory, routed depth reports, relation-conditioned SwiGLU, Deep Present recurrence, and a training-only multi-horizon Future Cone objective.
• A native 16,384-token byte-level BPE tokenizer, trained only on the official BabyLM 2026 Strict training split.
• Trained from scratch on a single NVIDIA RTX 3090 with BF16 autocast.

Verified intermediate results

BabyLM 2026 fast checkpoint evaluation at 500M words

The result bundle contains 18 files and passed a complete size and SHA-256 readback against its result manifest. This is a single-seed intermediate-checkpoint result and should be treated as a developmental measurement.

Context against published systems

The following table places the intermediate result beside published reference numbers. Evaluation suites and scopes changed across years; the table provides descriptive historical context.

ARO's clearest intermediate strength is grammatical minimal-pair accuracy. Leaderboard rank, downstream transfer, and component-level causal attribution will be assessed from the final evaluation and later controlled studies.

Final 1B evaluation

Architecture

ARO uses one full-width present state, h ∈ R^(B×T×768), throughout eleven physical layers. Each layer combines several complementary paths that specialize in exact retrieval, compressed memory, depth evidence, and relation-conditioned transformation.

token ids
   │
   ▼
16,384-way tied embedding × √768
   │
   ▼
11 × ARO block
   ├── causal local/global grouped-query attention
   ├── recurrent GatedDeltaNet memory
   ├── routed reports from earlier depth
   ├── relation-conditioned SwiGLU
   ├── ADS feature path
   └── bounded Present-state write
   │
   ▼
RMSNorm → tied vocabulary projection → next-token distribution

Causal local/global attention

Most layers use a 256-token causal window. Layers 3, 7, and 10 use global causal attention. The attention path has eight query heads, two key/value heads, and a head dimension of 64. RoPE supplies positional information.

Each layer also learns a query-dependent post-attention head gate and a shared residual 64×64 Value basis. The gate modulates the retrieved moments, while the Value basis provides a small learned coordinate transform shared across heads in that layer.

GatedDeltaNet memory

Attention is paired with a recurrent GatedDeltaNet path. It compresses the entire visible prefix into a continuously updated state, giving the model an all-past memory path alongside exact attention retrieval. The BabyLM configuration uses six memory heads of width 48, a 384-dimensional bottleneck, expansion factor 2, and a short convolution of width 4.

Deep Present

The attention calculation is refined for two internal loops. The transformed Value representation is held fixed while the current query and its headwise gate are renewed. This lets the model reconsider how the present state reads the same available evidence without introducing a second residual stream.

Routed depth reports

Starting at configured depths, a four-head router retrieves compact reports derived from earlier layer changes. These reports give later blocks selective access to depth history.

Relation-conditioned SwiGLU

The feed-forward path has width 2,752. Before the SwiGLU output is written back, the block constructs twelve relation modes from separately normalized evidence sources: the current state, attention output, recurrent-memory output, routed depth reports, displacement from the token embedding, and attention dispersion where available.

For present observation p and evidence source e_i, shared projections produce directional interactions:

F_i = L(p) ⊙ R(e_i)
R_i = R(p) ⊙ L(e_i)

Evidence weights are normalized across sources. Symmetric, antisymmetric, product, squared-difference, self, cross-source, and relative-magnitude features are combined into a twelve-dimensional relation state. A learned 12 → 4×2752 basis then modulates every SwiGLU gate and value unit independently. A bounded steering gain controls how the proposed block update reorients the present state.

ADS

The first 256 feed-forward units receive an additional adaptive feature path. ADS is independent of relation steering and supplies a compact specialized subspace inside the wider feed-forward computation.

Future Cone auxiliary objective

The primary objective is ordinary causal next-token prediction. During training, three lightweight predictor heads also encourage the current hidden state to retain information useful for near, middle, and farther future horizons. Projected future targets are detached; gradients flow through the current representation and predictor heads, not backward through future token states. The auxiliary weights are 0.02, 0.02, and 0.01, with a 40M-token ramp.

The Future Cone heads are training-only. They do not expose future tokens during inference and are excluded from the exported inference weights.

Configuration

Training data

ARO was trained only on the official detoxified English BabyLM 2026 Strict corpus at revision 9e57baaaa91ac3c638746be14d1d5fa6c789f4cf. The packed corpus contains exactly 100,000,000 whitespace-delimited words and 151,489,032 tokenizer tokens per pass.

Documents were packed in a deterministic sorted-source stream as BOS + source tokens + EOS. Both tokenizer training and model training used the released BabyLM training corpus exclusively.

Tokenizer

The tokenizer is a 16,384-entry byte-level BPE trained from scratch on the official training split. It follows the BabyLM 2026 baseline tokenizer family while using an ARO-native vocabulary size.

Tokenizer SHA-256:

8edfa792838717e0940c309880a71acf1b276d34981576212b0fffaf3cf1d909

Training recipe

The sequence-length curriculum kept the optimization batch constant while increasing context:

The learning-rate schedule was transferred from earlier ARO learning-curve evidence. It used a 40M-subword warmup, a 1e-3 plateau, gradual half-cosine transitions toward 8e-4 and 6e-4, and a terminal learning rate of 6e-4.

The idea: the Ultimate Present and dependent origination

The starting image for ARO was a description of time encountered in accounts of near-death experience: the observer stands at the present, the past extends behind, and the future opens ahead. Past, present, and future appear as one panorama centered on a single point of observation. “The Ultimate Present” is the name given to that center.

ARO keeps one full-rank Present coordinate at the center of the model. Causal attention retrieves precise visible evidence. GatedDeltaNet maintains a compressed history. Depth routing recovers useful earlier transformations. Relation-conditioned feed-forward computation asks how these sources agree, differ, and interact. The Future Cone objective shapes the current representation toward several prediction horizons during training.

Buddhist dependent origination supplied a second principle: the present is formed through relations among conditions. ARO's twelve learned relation modes take their width from the twelve links of dependent origination, while a shared lower-triangular operator and its transpose provide forward and reverse readings. The modes are learned latent coordinates; they are not hard-coded labels for the twelve Buddhist links.

Deep Present brings the two sources together. Its first pass forms the current state from past evidence. Its second pass holds the principal past-side evidence fixed, regenerates the query from the updated Present, and reads the same past again. A changed Present extracts a new interpretation from fixed past evidence.

The philosophical sources provided a coherent architectural question. ARO converts that question into executable mechanisms whose language-model performance can be measured experimentally.

Usage

This model uses custom Transformers code. Review the repository source before enabling remote code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AwakeningOS/ARO-BabyLM-1B"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Language is", return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Integrity and provenance

The Hugging Face adapter saves deterministic RoPE and causal-mask buffers because Transformers meta-device loading leaves custom non-persistent buffers uninitialized. The exported weight tensors exactly preserve the trained parameters.

Intended use

ARO-BabyLM-1B is a research base model for studying data-efficient language acquisition, small causal architectures, grammatical generalization, and learning curves under a fixed exposure budget. It has not been instruction-tuned, preference-tuned, or safety-tuned.

Limitations

• The model is English-only and trained on a small, developmentally motivated corpus.
• It is a base next-token model, not an instruction-following assistant.
• The current reported BabyLM result is from one seed and an intermediate fast checkpoint suite.
• Strong BLiMP performance does not by itself establish broad reasoning, factual knowledge, or generation quality.
• The architecture contains several interacting mechanisms; the present run measures the complete system and does not isolate the causal contribution of each mechanism.
• Training data can contain social, cultural, and historical biases inherited from dialogue, subtitles, books, and Wikipedia-derived text.

Evaluation references

• BabyLM 2026 evaluation pipeline
• BabyLM 2025 findings
• BabyLM 2024 evaluation pipeline
• BabyLM 2026 Strict dataset

License

ARO-BabyLM-1B is released under the Apache License 2.0.

Copyright 2026 AwakeningOS.

Citation

A formal technical report and citation entry will be added after the complete evaluation package is frozen.