BabyLM2026-Strict-Small-looped_GPT-BERT

Looped GPT-BERT 4x12 for BabyLM 2026 Strict-Small

Looped GPT-BERT 4x12 is a 12.18M-parameter English language model trained from scratch for the BabyLM 2026 Strict-Small track. It combines the hybrid causal/masked training objective of GPT-BERT with a looped Transformer backbone: four physical Transformer layers are reused for 12 loop iterations. This gives 48 sequential Transformer-layer applications while retaining only four unique physical layers.

The final model uses an 8,192-token byte-level BPE tokenizer trained on the training corpus, a 1:3 masked-to-causal objective ratio, and a maximum pretraining sequence length of 128. It was trained for 10 corpus passes over 7,482,193 words. Intermediate word-milestone checkpoints are available as Hugging Face revisions for BabyLM fast evaluation.

This repository contains custom Transformers code. Load the model with trust_remote_code=True after reviewing configuration_gpt_bert.py and modeling_gpt_bert.py.

Model at a glance

Architecture

Looped backbone

Let x denote the static token embeddings and z_t the recurrent state after loop t. The backbone follows the additive loop update

z_0     = 0
z_{t+1} = F_theta(x + z_t),  t = 0, ..., 11

where F_theta is a stack of four physical Transformer layers. The same parameters theta are reused on every loop. The model therefore increases sequential training and inference compute without multiplying its static parameter count by 12.

Each physical layer contains multi-head self-attention, a feed-forward block, residual connections, layer normalization, relative-position embeddings, and Dense Weighted Average (DWA) residual-state mixing. The token embedding matrix is tied to the output language-model head.

The loop is implemented as follows:

1. Static token embeddings are computed once.
2. The first loop starts from an all-zero recurrent state.
3. At the beginning of every loop, the recurrent state is added to the static token embeddings.
4. The resulting state passes through all four physical layers.
5. The final state becomes the recurrent input to the next loop.
6. Logits are produced from the state after loop 12.

This is not a 48-layer model with 48 independently parameterized layers. It is a four-layer model whose weights are applied 12 times.

DWA scope

This checkpoint uses dwa_scope="physical_layer". Each physical layer owns a two-block DWA module covering that layer's attention and feed-forward updates. Its accumulator is reinitialized when entering that physical layer on every loop. Consequently:

• DWA mixes states inside one physical layer.
• DWA does not accumulate states across different physical layers.
• DWA does not carry its accumulator across loop boundaries.
• The physical layer and its DWA parameters are still shared across all 12 loop iterations.

The recurrent z_t state carries information across loops; the DWA accumulator does not.

Hybrid GPT-BERT objective

The model is trained using one shared backbone and one tied prediction head, but with two attention and target constructions:

• Causal language modeling (75% of examples): attention is lower triangular within each packed segment, and the model predicts the next non-padding token.
• Masked next-token prediction (25% of examples): attention is bidirectional within each packed segment. Only labels associated with selected masked tokens contribute to cross-entropy; all other labels are set to -100 and ignored.

The optimized objective is

L = 0.25 * L_MNTP + 0.75 * L_CLM + 1e-4 * L_z

where L_z is a log-partition regularizer. The BERT/GPT ratio is enforced both in per-device batch composition and in the loss weighting. With 32 sequences per GPU, each rank receives 8 masked-objective rows and 24 causal rows.

The same uploaded weights can be instantiated through either AutoModelForCausalLM or AutoModelForMaskedLM. These are two attention-mode interfaces over the same hybrid-trained checkpoint, not two separately trained models.

Tokenizer

The model uses a custom PreTrainedTokenizerFast byte-level BPE tokenizer trained only on the local BabyLM English training corpus.

Special tokens are:

An </s> token is appended to every non-empty source line before segmentation. This explicitly trains the causal objective to predict sequence termination. Packed segments begin with <s> and use block-diagonal attention masks so that tokens cannot attend across segment boundaries.

Training data

The model was trained on a cleaned English copy of the BabyLM 2026 Strict-Small dataset. The local corpus contains 7,482,193 whitespace-delimited words, below the 10M-word Strict-Small limit. Non-English strings were filtered during local preprocessing, and non-empty source lines were treated as document or utterance boundaries.

The run used 10 corpus passes, for approximately 74,821,930 raw-word exposures. This is distinct from the number of unique words: the unique training corpus is 7.48M words and is revisited over 10 epochs. No teacher model, preference data, external pretrained checkpoint, or synthetic-data augmentation was used.

Training procedure

The target mask probability for masked-objective examples decreases linearly from 0.30 to 0.15 over the full 4,220-step run. Of selected masked targets, the corruption policy uses a 0.10 random-token probability and a 0.10 unchanged-token probability; the remaining selected targets use <mask>. Special token IDs are excluded from masking.

An exponential moving average was maintained during training. The Hugging Face -hf exports and the revisions in this repository contain the regular online model weights, not the separately saved _ema.bin state.

Training-curve summary

The nearest logged point to the final step is step 4,200:

The best logged hybrid loss was 3.3248 at step 3,500, with a hybrid token accuracy of 37.65. These training losses combine two objectives and should not be interpreted as a standalone causal perplexity.

Intermediate checkpoints

The main revision contains the final epoch-10 model. BabyLM word-milestone checkpoints are stored as Git revisions:

chck_1M, chck_2M, chck_3M, chck_4M, chck_5M,
chck_6M, chck_7M, chck_8M, chck_9M, chck_10M,
chck_20M, chck_30M, chck_40M, chck_50M, chck_60M, chck_70M

Milestones count total raw whitespace-delimited words consumed, not unique words and not BPE tokens. Because a checkpoint is saved after completing the batch that crosses a milestone, its exact consumed-word count can be slightly above the revision label. The run ends after approximately 74.82M word exposures, so no chck_80M or later revision exists.

Each revision is intended to be self-contained and includes the model configuration, custom model code, tokenizer files, special-token mapping, and weights.

Usage

Requirements

The model was tested with Python 3.11, PyTorch, and Transformers 4.51.3. A minimal installation is:

pip install "transformers==4.51.3" torch

Load the final causal model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision="main",
    trust_remote_code=True,
).to(device).eval()

text = "Once upon a time"
inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)

print(outputs.logits.shape)

Generate text

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
).to(device).eval()

prompt = "The little girl opened the door and"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    generated = model.generate(
        **inputs,
        max_new_tokens=32,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.05,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(generated[0], skip_special_tokens=True))

The model does not implement a KV cache. Generation recomputes the full prefix at every decoding step and is therefore slower than generation from a similarly sized decoder model with caching. For behavior closest to pretraining, keep the combined prompt and generated sequence at or below 128 tokens. Although the configuration permits up to 512 positions, positions beyond 128 were not seen during pretraining.

Use the masked-language-model interface

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
).to(device).eval()

text = f"The child {tokenizer.mask_token} the ball."
inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
    logits = model(**inputs).logits

mask_position = (inputs.input_ids[0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0]
top_ids = logits[0, mask_position].topk(5, dim=-1).indices[0]
print(tokenizer.convert_ids_to_tokens(top_ids.tolist()))

Load an intermediate revision

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT"
revision = "chck_10M"

tokenizer = AutoTokenizer.from_pretrained(repo_id, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
).eval()

Evaluation

The final model was evaluated locally with the official BabyLM 2026 evaluation repository using the causal backend. The numbers below are self-reported development results from the local evaluation artifacts. They are not a substitute for the server-side leaderboard score produced from the submitted prediction file.

Zero-shot results

EWoK, reading, AoA, and any other tasks not listed above are not reported in this card. The exact evaluation-repository commit should be recorded before these numbers are used in a paper, since evaluation code and task aggregation can change during an active challenge.

Fine-tuning results

All values are percentages. The task metrics should not be averaged directly without following the official BabyLM aggregation procedure.

Reproduce evaluation

From the strict directory of the BabyLM evaluation repository:

MODEL_ID="Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT"

# Full zero-shot evaluation of the final model.
bash scripts/eval_zero_shot.sh \
  "$MODEL_ID" causal evaluation_data/full_eval

# Fine-tuning evaluation of the final model.
bash scripts/eval_finetuning.sh \
  --model_path "$MODEL_ID"

# Fast evaluation of one intermediate checkpoint.
bash scripts/eval_zero_shot_fast.sh \
  "$MODEL_ID" chck_10M causal evaluation_data/fast_eval

After full evaluation of main and fast evaluation of all available checkpoint revisions, collate the Strict-Small prediction file with the command expected by the checked-out evaluation version:

bash scripts/collate_preds.sh \
  "$MODEL_ID" causal strict-small

Reproducing pretraining

The local training command is:

torchrun --nproc_per_node=8 --standalone \
  scripts/gpt_bert_training/train_gpt_bert_10m_trainer.py \
  --train-config configs/8k_ratio1to3_lr0141_loop4x12_decay2600_eos_submission.yaml

The training configuration saves ordinary epoch/state checkpoints in the main output directory and BabyLM word-milestone checkpoints beneath milestone_checkpoints/chck_<N>M/. Each milestone contains a Hugging Face export used to populate the corresponding repository revision.

Compute and efficiency

• Static model size: 12,181,780 parameters.
• Sequential backbone use: four physical layers applied over 12 loops.
• Distributed training: 8 GPUs.
• Reported wall-clock time for the final run: approximately 15-20 minutes.
• Approximate aggregate training time: 2.0-2.7 GPU-hours, computed as wall time multiplied by 8 GPUs.
• A loop-aware 6NT-style estimate is approximately 90.9 training PFLOPs when recurrent block parameters are counted on every application.

The FLOP value is an analytical approximation, not a hardware-profiler measurement. Applying 6 * static_parameter_count * token_count without accounting for the 12 loop applications underestimates this architecture's compute. Conversely, multiplying every parameter by 12 overestimates compute because embeddings and the output head are not reapplied in the same way as the recurrent physical layers. Hardware model, utilization, communication overhead, and exact kernel FLOPs are not captured by the estimate.

Intended uses

This model is intended for:

• Research on data-efficient language-model pretraining.
• Studying recurrent or looped Transformers under a fixed parameter budget.
• Comparing causal and masked interfaces from a shared hybrid checkpoint.
• BabyLM 2026 Strict-Small evaluation and checkpoint learning-curve analysis.
• Small-scale English continuation, representation, and linguistic probing experiments.

This model is not instruction-tuned, chat-tuned, preference-aligned, or designed for production deployment.

Limitations and risks

• Small and narrow corpus: the model saw only 7.48M unique training words and is English-only.
• Single training seed: results may include seed variance and should not be treated as a robust architecture average.
• Short trained context: pretraining used 128-token sequences even though the configuration supports 512 positions.
• No KV cache: autoregressive generation is compute-intensive because all 12 loops are rerun over the full prefix for every generated token.
• No instruction alignment: prompts are treated as text continuation, not as instructions that the model is guaranteed to follow.
• Potential harmful output: the model may reproduce bias, stereotypes, private-looking strings, offensive content, or factual errors present in or inferred from its training data.
• Limited factual reliability: this model should not be used for medical, legal, financial, safety-critical, or other high-stakes decisions.
• Custom code execution: trust_remote_code=True executes Python files from the selected repository revision. Review and pin the revision or commit hash in security-sensitive environments.
• Incomplete evaluation record: only the tasks listed above are currently documented, and leaderboard scores may differ from local development scores.

Repository structure and custom code

The Hugging Face revisions include:

• config.json: model dimensions, loop count, DWA scope, architecture name, and Transformers auto_map entries.
• configuration_gpt_bert.py: custom PretrainedConfig implementation.
• modeling_gpt_bert.py: shared backbone plus causal and masked LM wrappers.
• tokenizer.json: the 8k byte-level BPE tokenizer.
• tokenizer_config.json and special_tokens_map.json: tokenizer metadata.
• Model weights and generation configuration.

The auto_map entries tell Transformers which repository files and classes to load:

{
  "AutoConfig": "configuration_gpt_bert.ModelConfig",
  "AutoModel": "modeling_gpt_bert.GPTBERT",
  "AutoModelForCausalLM": "modeling_gpt_bert.GPTBERTForCausalLM",
  "AutoModelForMaskedLM": "modeling_gpt_bert.GPTBERTForMaskedLM"
}

License

No project-level license was present in the training repository when this model card was generated. An explicit model-and-code license should be added only after checking compatibility with the upstream GPT-BERT implementation, the BabyLM dataset terms, and any other incorporated code. Until a license is added, this model card does not grant additional reuse rights.

Citation

A formal paper citation for this specific model is not yet available. Until the challenge paper is released, the model repository can be cited as:

@misc{survivor613_looped_gptbert_2026,
  author       = {Survivor613},
  title        = {Looped GPT-BERT 4x12 for BabyLM 2026 Strict-Small},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT}
}

Please also cite the original GPT-BERT work:

@inproceedings{charpentier-samuel-2024-bert,
  title     = {{GPT} or {BERT}: why not both?},
  author    = {Charpentier, Lucas Georges Gabriel and Samuel, David},
  booktitle = {The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning},
  year      = {2024},
  publisher = {Association for Computational Linguistics},
  pages     = {262--283},
  url       = {https://aclanthology.org/2024.conll-babylm.24/}
}

For the 2026 challenge setup, cite:

@misc{choshen2026babylmturns4goes,
  title         = {BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop},
  author        = {Leshem Choshen and Ryan Cotterell and Mustafa Omer Gul and Jaap Jumelet and Tal Linzen and Aaron Mueller and Suchir Salhan and Raj Sanjay Shah and Alex Warstadt and Ethan Gotlieb Wilcox},
  year          = {2026},
  eprint        = {2602.20092},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2602.20092}
}

References and acknowledgements

• GPT or BERT: why not both? introduced the shared causal/masked GPT-BERT training framework.
• LTG-BERT is the architectural predecessor cited by GPT-BERT.
• Leiay/looped_transformer informed the additive recurrent update used in this experiment.
• BabyLM 2026 evaluation provides the official evaluation and submission pipeline.
• BabyLM-community GPT-BERT Strict-Small baseline provides the upstream Hugging Face custom-code model format used as a reference.