mosaic-memory-d384-bbpe16k

Noisy auxiliary checkpoint

This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.

Model and data

Training operation

On one quarter of batch rows, an extra causal forward encodes the complete prefix. A learned scalar precision at each position controls Gaussian noise in the hidden state. The corrupted state predicts the next word-starting token, and a squared penalty keeps mean precision near 0.35. The auxiliary loss has weight 0.15. The primary bidirectional masked LM retains its complete context.

The implemented operation is a noisy causal auxiliary objective. The exported model has no bounded-memory architecture, and the run does not identify a human memory mechanism.

Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "miguelcsx/mosaic-memory-d384-bbpe16k"

tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    revision="main",
    trust_remote_code=True,
)

main and chck_100M contain the final checkpoint. Earlier checkpoints are stored as chck_1M through chck_10M, then chck_20M through chck_90M.

Remote code is required for the custom inference class. Review tolm.py before loading it in a security-sensitive environment. The canonical code is maintained at github.com/miguelcsx/tolm.

Evaluation and provenance

All required checkpoint revisions were evaluated with the same BabyLM fast suite used for the controlled family. These outputs are local development evidence, not a leaderboard rank or a superiority claim. Machine-readable configuration and file provenance are recorded in training_manifest.json and release_manifest.json.

Limitations

• This is one trained run, so it does not estimate training-run variance.
• The model is a small English masked LM, not a general-purpose assistant.
• The recipe label describes an implemented operation. It does not establish child-like memory or a causal claim about children.