neurobert-cog-v2

neurobert-cog-v2

ChineseBabyLM 2026 Cognitive Modeling Track submission model.

Model

neurobert-cog-v2 is a small Chinese BERT-style encoder trained from random initialization on the official ChineseBabyLM Chinese corpus, then exported with a lightweight brain-alignment representation adapter.

• Architecture: BERT-style encoder
• Layers: 10
• Hidden size: 384
• Attention heads: 8
• Intermediate size: 1536
• Maximum sequence length: 192
• Tokenizer: raw Chinese BPE
• Vocabulary size: 32000
• Adapter layers: [-7, -5, -3, -1]
• Adapter scale: 0.02

The exported model exposes the adapter output as last_hidden_state, so the official CogBench pipeline can load it through:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("chinese-babylm-ws/neurobert-cog-v2", trust_remote_code=True)
model = AutoModel.from_pretrained("chinese-babylm-ws/neurobert-cog-v2", trust_remote_code=True)

Training Data

Pretraining used a cleaned subset of the official chinese-babylm-org/babylm-zho-100M corpus.

• Jieba 0.42.1 tokens: 81,062,469
• Official metadata num-tokens sum: 97,580,273
• Official baseline weights were not used
• LLM-generated or distilled data was not used

The representation adapter was trained with public CogBench train/dev story-level and word-level fMRI supervision. Hidden test labels were not used.

Reproducibility

Reproduction materials are provided in repro/, including training scripts, manifests, final-pipeline config, and environment requirements. The official training corpus and CogBench data are not redistributed in this repository.