gpt2_ParFindFast_eng_nl_zho

BabyLM 2026 - Multilingual GPT-2 (ParadigmFinder)

Track: BabyLM 2026 Multilingual
Architecture: GPT-2
Tokenizer: ParadigmFinder with boundary discovery support

A multilingual GPT-2 language model trained for the BabyLM 2026 multilingual setting on English, Dutch, and Chinese. This release is intended as the ParadigmFinder counterpart to the multilingual MoP baseline: it uses the same model family and the same multilingual_selection training data for language-model training, while replacing the tokenizer with a ParadigmFinder-based segmentation pipeline.

The tokenizer and model were exported together and can be loaded directly from the Hub. Since the tokenizer implementation is custom, loading requires trust_remote_code=True.

---

Model Details

Architecture

Tokenizer

The model uses a custom ParadigmFinder tokenizer exported as EnhancedParadigmTokenizerWrapper. In this multilingual configuration, tokenizer training uses a soft per-language vocabulary budget and boundary discovery support designed to remain compatible with space-free scripts such as Chinese.

Tokenizer highlights:

• Boundary discovery enabled: true
• Boundary discovery mode: space_free_only
• Shorter-first sentence ordering: true
• Explicit word-boundary insertion during tokenization: disabled (use_word_boundaries=false)
• Vocabulary budget: 16,384 token slots per language
• Languages: English (eng), Dutch (nld), Chinese (zho)

Special tokens:

• <pad> = 0
• <unk> = 1
• <s> = 2
• </s> = 3

This repo includes the tokenizer code and supporting files needed for loading:

• tokenizer.py
• boundary_discovery.py
• preprocessing.py
• paradigm_utils.py
• paradigms.json
• multilingual_meta.json

Training Data

This model was trained on the BabyLM multilingual data selection stored locally as multilingual_selection, with the three languages:

• English (eng)
• Dutch (nld)
• Chinese (zho)

For comparability with the multilingual MoP baseline, the GPT-2 model training uses the same multilingual corpus selection. In this experiment line, tokenizer training and model training were aligned to the same multilingual data source.

Training Procedure

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "achille-fusco/gpt2_ParFindFast_eng_nl_zho"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

prompt_en = "The child looked at"
inputs = tokenizer(prompt_en, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_nl = "Het kind keek naar"
inputs = tokenizer(prompt_nl, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_zh = "孩子看着"
inputs = tokenizer(prompt_zh, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

If the environment has a read-only Hugging Face cache, set a writable cache first:

export HF_HOME=/tmp/hf_cache

---

Evaluation

This model was evaluated with the BabyLM 2026 multilingual evaluation pipeline across zero-shot and fine-tuned tasks.

Summary scores:

Selected zero-shot results:

These numbers correspond to the collated submission artifact produced for this export.

---

Limitations

• This is a small multilingual BabyLM-scale model and is not intended as a production general-purpose LM.
• The tokenizer relies on custom remote code, so some environments may require extra care when loading.
• No explicit language identifier is used; mixed-language prompting may lead to unstable behavior.
• Compared with the multilingual MoP baseline, this model is stronger on some Chinese and compositional zero-shot evaluations, but weaker overall after finetuning.

---

Citation

If you use this model, please cite the model repository and the BabyLM shared task.

@misc{fusco2026parfindfast,
  author       = {Fusco, Achille},
  title        = {{BabyLM 2026 Multilingual GPT-2 (ParadigmFinder)}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/achille-fusco/gpt2_ParFindFast_eng_nl_zho}},
  note         = {BabyLM 2026 multilingual model release with ParadigmFinder tokenizer.}
}