gpt2_parfind_en_nl_zh_equal

gpt2_parfind_en_nl_zh_equal

Multilingual GPT-2 model for the BabyLM 2026 multilingual setting, paired with a custom ParadigmFinder tokenizer trained on English, Dutch, and Chinese.

This export comes from:

• tokenizer experiment: toktrain_ParFindFast_ML_BD_budget16k_zhchildes_v1.0
• model experiment: model_gpt2_ParFindFast_ML_BD_budget16k_zhchildes_v1.0

The model and tokenizer are packaged together for direct Hugging Face upload. Because the tokenizer uses custom code, loading requires trust_remote_code=True.

Model

• architecture: GPT2LMHeadModel
• layers: 12
• attention heads: 12
• hidden size: 768
• max positions: 1024
• training sequence length: 512
• exported model vocab size: 44090

Tokenizer

• class: EnhancedParadigmTokenizerWrapper
• paradigm family: ParadigmFinder
• boundary discovery: true
• boundary discovery mode: space_free_only
• multilingual budgeting: soft merge
• budget per language: 16384
• languages: eng, nld, zho

This tokenizer includes the custom runtime files needed for HF loading:

• tokenizer.py
• boundary_discovery.py
• preprocessing.py
• paradigm_utils.py
• paradigms.json
• multilingual_meta.json

Training Data

The model was trained on the BabyLM 2026 multilingual corpus selection used in this repo, with aligned tokenizer training and LM training over the same multilingual data source.

Languages:

• English
• Dutch
• Chinese

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "NeTSlab/gpt2_parfind_en_nl_zh_equal"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "The child looked at"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

If your Hugging Face cache is not writable, set:

export HF_HOME=/tmp/hf_cache

Notes

• This is a BabyLM-scale research model, not a production general-purpose LM.
• The tokenizer uses custom remote code, so some environments may need trust_remote_code=True.
• The multilingual vocabulary budget is equal per language at configuration time, but the final merged vocabulary is derived after deduplication.