gpt-bert-entmax-graded

gpt-bert-entmax-graded

BabyLM 2026 Strict-Small submission (NTUA). A GPT-BERT hybrid language model: gpt_bert_mixed — hidden 384, 12 layers, 6 heads, alpha-entmax attention with per-layer alpha schedule [1.25, 1.25, 1.25, 1.25, 1.75, 1.75, 1.75, 1.75, 1.5, 1.5, 1.5, 1.5].

• Training data: BabyLM-community/BabyLM-2026-Strict-Small (≤100M words, counting repeated exposures, per CfP §4.2). Uniform ~10 passes via bin-packing (each segment once per epoch).
• Objective: mixed masked+causal LM (causal_prob 0.0625, masked-heavy). Evaluate with the mntp backend.
• Intermediate checkpoints: available as branches chck_1M … chck_100M (words seen), per the BabyLM checkpoint convention.

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)

Custom architecture — trust_remote_code=True is required (config auto_map points at modeling_gpt_bert). Requires pip install entmax (entmax attention). Trained with the NTUA BabyLM pipeline (https://github.com/emhadzi/BabyLM).