hawk_mopbf_en_nl_zh_equal

NeTS - hawk_mopbf_en_nl_zh_equal

A multilingual (English / Dutch / Chinese) Hawk language model trained for the BabyLM 2026 Multilingual track (EMNLP 2026). This is a baseline model in the NeTS-lab BabyLM-2026 series: it pairs a recurrent Hawk (RG-LRU) backbone with the morphologically-aware MorPiece (MoP) tokenizer (+ byte_fallback), trained on the three target languages (eng / nld / zho) with a byte-premium-balanced word budget under the baseline regimen.

It is the Hawk counterpart to the Transformer baseline NeTS-lab/babylm26_multiling_gpt2_MoP16K_baseline; the two share the same tokenizer and the same training data, so the comparison isolates the architecture (linear-recurrent vs. attention).

Note on baselines. These models are released as controlled reference points for the NeTS-lab BabyLM-2026 study. Their purpose is a clean, matched comparison across architectures and tokenizers — not leaderboard maximisation.

---

Model details

• Developed by: NeTS Lab, IUSS Pavia (with Claude Opus 4.8 fixes and specific HPC optimizations)
• Model type: Decoder-only causal LM, recurrent (Hawk / RG-LRU) — no global self-attention
• Languages: English (eng), Dutch (nld), Chinese (zho)
• Tokenizer: MorPiece (MoP) + byte_fallback, ~39.7K vocabulary, shared multilingual
• License: MIT
• Sibling models: *_gpt2_MoP16K_baseline (Transformer), and the eMG / SSM-eMG variants

Architecture

Hawk is the gated linear-recurrent backbone from the Griffin family (De et al., 2024). Each block uses a Real-Gated Linear Recurrent Unit (RG-LRU) in place of attention, combined with a gated MLP and RMSNorm. Loading requires trust_remote_code=True because the hawk_rglru block is provided via custom modelling code.

The model is a pure Hawk recurrent stack (RG-LRU + depthwise conv + gated MLP, RMSNorm pre-norm). There is no self-attention, so the leaderboard "attention heads" field is -1. Parameter count is with tied embeddings (lm_head shares wte). Note that Hawk has no learned position embeddings — max_position_embeddings is a config field, not a hard context limit.

Tokenizer — MorPiece (MoP)

MorPiece is a split-based tokenizer that incrementally segments words into candidate morphemes by applying Yang's (2016) Tolerance Principle at every character as a word traverses a dual root/inflection trie. Splits are licensed only when the TP holds bilaterally (root trie and inflection trie). The result is a morphology-aware vocabulary motivated by developmental linguistics rather than pure frequency. For this multilingual experiment we used the --boundary-discovery option to ignore whitespaces and process zho the same way of eng and nld.

• Shared across all three languages (single multilingual MoP tokenizer)
• Exported in HuggingFace WordPiece format with ++ continuing-subword prefix
• Actual vocabulary: 39,697 tokens (the MoP16K in the original repo name (NeTS-lab/babylm26_multiling_hawk_MoP16K_baseline) indicates that a maximum of 16K tokens per language can be stored in the final lexicon;
• the only difference with this tokenizer is the inclusion of byte_fallback to)

See the MorPiece repository for details: https://github.com/cristianochesi/morpiece

---

Training data

Official BabyLM 2026 Multilingual data for eng / nld / zho. Languages are drawn in a byte-premium round-robin during training, and the save-point milestones are denominated in byte-premium-adjusted English-equivalent words (BP: eng 1.000, nld 1.0516, zho 0.9360), per the multilingual track's word budget. Training regimen: baseline. Per-corpus sizes: eng 56.2M / nld 57.0M / zho 50.0M model tokens (≈34.2M English-equivalent words each ≈ 102.6M total/epoch).

No custom corpus, no synthetic augmentation, no human-annotated preference data. A cleaning procedure stripped metalinguistic information (e.g. tiers). Preprocessing routine can be found here: https://github.com/cristianochesi/babylm-2026/tree/main/01-preprocess

Training procedure

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "NeTSlab/hawk_mopbf_en_nl_zh_equal"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

text = "the cats are"
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required to load the custom hawk_rglru block and tokenizer. The repo must contain modeling_hawk.py alongside config.json (the auto_map points to it). Generation is correct but has no KV cache — the recurrent backbone recomputes the full prefix each step, so generate() is O(T) per token.

---

Evaluation

Evaluated with the BabyLM 2026 multilingual harness (lm-eval-style), including BLiMP, MultiBLiMP (Dutch), and SIGMORPHON 2022 morphology, alongside the official multilingual benchmarks.

---

Intended use & limitations

A small, sample-efficient research LM for studying cognitively-plausible language modelling under a constrained (developmentally motivated) data budget. It is not intended for production use. As a baseline trained on a limited multilingual corpus, outputs are not reliable for downstream generation and may reflect biases in the training data.

Citation

@misc{chesi2026babylm_hawk_mop,
  title  = {Multilingual Hawk + MorPiece baseline for BabyLM 2026},
  author = {Chesi, Cristiano and {NeTS Lab, IUSS Pavia}},
  year   = {2026},
  note   = {BabyLM 2026 Multilingual track baseline},
  howpublished = {\url{https://huggingface.co/NeTS-lab/babylm26_multiling_hawk_MoP16K_baseline}}
}

Contact: [email protected] · NeTS Lab, IUSS Pavia