instanton-hybrid-dialogue

instanton-hybrid-dialogue

A 40.2M-parameter GPT-BERT masked–causal hybrid trained entirely from scratch on a dialogue-heavy remix of the BabyLM 2026 Strict corpus, within the Strict-Small budget (≤10M words).

Part of the instanton family, submitted to the BabyLM 2026 Challenge. It is the mixture-variant counterpart to qyxu1994/instanton-hybrid, which uses the official mixture; architecture and recipe are identical, so the pair isolates the effect of the data mixture.

Training data

Resampled from the official detoxified BabyLM 2026 Strict (100M) pool — no external data — by drawing contiguous 100-line blocks without replacement (seeded) until per-source word targets were met. The mixture upweights spoken and conversational text (~6.2M words from OpenSubtitles, BNC, Switchboard and CHILDES), with children's books and simple encyclopedia making up the rest.

The research question: under a fixed 10M-word budget, does upweighting cognitively motivated, child-directed and conversational data improve sample efficiency?

Training objective and recipe

50/50 masked/causal blend after Charpentier & Samuel (2024), reproduced in-process on a single GPU (half the micro-batches of each optimizer step masked, half causal). LAMB, max lr 0.007, cosine with 1.6% warmup / 1.6% cooldown, weight decay 0.1, grad clip 2.0, z-loss 1e-4, mask ratio 0.30→0.15, sequence length 128→256→512, 16,384 tokens/step, seed 42, bf16. ~2 hours on one A100-80GB.

Checkpoints

Nineteen word-exposure revisions as git branches: chck_1M … chck_9M, chck_10M, chck_20M … chck_100M, plus main. Load with revision=.

Usage

Requires trust_remote_code=True (vendored modelling wrapper) and transformers 4.51.x; 5.x is not supported.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tok = AutoTokenizer.from_pretrained(
    "qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    "qyxu1994/instanton-hybrid-dialogue", trust_remote_code=True)

Score as a masked LM (pseudo-log-likelihood / mntp).

Evaluation

The dialogue-heavy mixture wins most zero-shot columns, GLUE and reading, and notably recovers the EWoK and entity-tracking ground the official-mixture hybrid gives up — evidence that the architecture and the mixture compose rather than compete. It nonetheless has the lower leaderboard Overall, because of the Global-PIQA result below.

Known limitations

Global-PIQA is below chance and unexplained. This model scores 16.5% on the 4-choice parallel subset (chance 25%) and 42% on the 2-choice non-parallel subset (chance 50%) — it systematically prefers the wrong solution. This is not a scoring artifact: the same code reproduces the official leaderboard score exactly for a sibling model. Nor is it the mixture alone: a causal GPT-2 trained on the same dialogue-heavy corpus scores 38.12. It appears specific to this model and we have no explanation for it. Treat this model's physical-commonsense behaviour as unreliable.

Age of acquisition is significantly negative (r = −0.165, p = 0.008), the most negative in the instanton family. Upweighting child-directed and conversational text moved this metric in the wrong direction, which is a negative result worth reporting: feeding a model more child-like text does not make its acquisition trajectory more child-like.

Intended use

A research artifact for studying sample-efficient language acquisition and data mixture effects. Trained on <10M words; not suitable for deployment. See the Global-PIQA limitation above before using it for anything commonsense-related.

Citation

Described in the accompanying BabyLM 2026 report, Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining. Architecture from Charpentier & Samuel (2024).