Downloads · 30 days
1.6K
15% of all-time downloads
l0ulan/chinese-babylm-nanochat-d22-step10000
chinese-babylm-nanochat-d22-step10000 is a text generation model from l0ulan. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
chinese-babylm-nanochat-d22-step10000 is a Chinese base causal language model trained with the karpathy/nanochat codebase for the NLPCC 2026 Chinese BabyLM shared task.
Downloads · 30 days
1.6K
15% of all-time downloads
All-time downloads
10.7K
Public
Repo size
3.4 GB
Likes
0
Public
Click a slice to open those files.
.bin3.4 GB · 100%
From the Hugging Face model README
chinese-babylm-nanochat-d22-step10000 is a Chinese base causal language model trained with the karpathy/nanochat codebase for the NLPCC 2026 Chinese BabyLM shared task.
This is a base continuation model, not an instruction-tuned or chat-aligned assistant. It should be prompted with plain text and interpreted as next-token continuation. It has not gone through SFT, RLHF, DPO, or safety alignment.
This checkpoint is intended for the Chinese BabyLM Challenge final model submission.
| Requirement | Status |
|---|---|
| One submitted model | This repo contains a single HF-exported checkpoint. |
| From scratch | Trained from randomly initialized weights. |
| No pretrained checkpoint | No pretrained model weights were loaded. |
| No distillation | No teacher-model distillation was used. |
| Data option | Official corpus option. |
| Pretraining data | chinese-babylm-org/babylm-zho-100M. |
| Evaluation leakage | No evaluation datasets were intentionally included in pretraining. |
| Frozen submission | Intended checkpoint is step 10000. |
Final submission repositories for the challenge should be public before the organizer deadline.
| Field | Value |
|---|---|
| Model type | Decoder-only causal language model |
| Architecture | NanoChat NanoChatForCausalLM |
| Training objective | Next-token prediction |
| Language | Chinese |
| Checkpoint | zh-d22-2gpu, step 10000 |
| Context length | 1024 tokens |
| Vocabulary size | 32768 |
| Tokenizer | Custom Chinese tokenizer trained on Chinese BabyLM text |
| Precision | bfloat16 |
| Export format | HuggingFace trust_remote_code |
The model configuration:
| Parameter | Value |
|---|---|
sequence_len | 1024 |
vocab_size | 32768 |
n_layer | 22 |
n_head | 22 |
n_kv_head | 22 |
n_embd | 1408 |
| Head dimension | 64 |
window_pattern | L |
representation_layer | 8 |
representation_hidden_states_mode | selected_only |
| Estimated total parameters | ~1.123B |
This repository includes custom model and tokenizer code:
configuration_nanochat.py
modeling_nanochat.py
tokenization_nanochat.py
Loaders must use trust_remote_code=True.
Expected files in this model repository:
config.json
pytorch_model.bin
tokenizer.pkl
tokenizer_config.json
special_tokens_map.json
configuration_nanochat.py
modeling_nanochat.py
tokenization_nanochat.py
README.md
.gitattributes
img/baseline_task_comparison.png
pytorch_model.bin is the model weight file. tokenizer.pkl is required by the custom tokenizer.
The model was trained on the official Chinese BabyLM corpus:
chinese-babylm-org/babylm-zho-100M
The corpus was converted into nanochat parquet shards:
| Split | Rows |
|---|---|
| Train | 182,680 |
| Validation | 918 |
Total prepared shards: 17.
No extra instruction data was used for this checkpoint.
A 32K Chinese tokenizer was trained from the Chinese BabyLM training data.
Tokenizer compression on Chinese BabyLM text:
| Split | This tokenizer bytes/token | GPT-2 bytes/token | GPT-4/cl100k bytes/token |
|---|---|---|---|
| Train | 4.98 | 1.43 | 2.20 |
| Validation | 4.81 | 1.44 | 2.19 |
This tokenizer is specialized for Chinese BabyLM data. It gives much better compression on the target corpus than GPT-2 or GPT-4 tokenizers, at the cost of weaker general-purpose compression on English, code, math, and mixed-domain text.
The final recovered training run used one RTX 5090 GPU.
Main training configuration:
| Field | Value |
|---|---|
| Depth | 22 |
| Sequence length | 1024 |
| Device batch size | 4 |
| Total batch size | 65536 tokens |
| Iterations | 10000 |
| Trained tokens | 655,360,000 |
| Dtype | bfloat16 |
| Final train loss | 0.015389 |
| Minimum observed train loss | 0.014984 |
| Total training time | 363.75 minutes |
| Peak memory | ~19.1 GiB |
The model was exported to HuggingFace format and evaluated with a fresh copy of the official Chinese BabyLM eval pipeline:
SiyuanSong2004/chinese-babylm-eval-pipeline
The final official run used the unmodified official pipeline with backend: causal.
The Hanzi tasks use the refreshed June 6 evaluation data/rules released by
the Chinese BabyLM organizers.
For cognitive evaluation, this repository exposes layer 8 as the HF AutoModel
representation (representation_layer=8, representation_hidden_states_mode=selected_only).
The language-model weights are still the same d22 step10000 checkpoint.
The repository root includes config.yaml, which freezes the official
evaluation pipeline settings and NLU fine-tuning hyperparameters for hidden
evaluation. It is an evaluation config, not a model architecture or training
config.
Final gathered official open-task scores:
| Task | Score |
|---|---|
zhoblimp | 67.45 |
hanzi_structure | 52.35 |
hanzi_pinyin | 51.00 |
word_fmri | 55.80 |
fmri | 10.66 |
afqmc | 68.61 |
ocnli | 55.59 |
tnews | 51.31 |
cluewsc2020 | 59.54 |
| Mean over open tasks | 52.48 |
The figure compares this model against three representative baselines:
bert-base-chinese: strong pretrained encoder baseline.Qwen3-0.6B: strong generative baseline.babylm-chinese-bert-14m-epoch20: strong small BabyLM-trained baseline.Our model is shown with the darkest pink. Its Hanzi scores use the June 6 refreshed evaluation data; baseline values are the public reference scores listed in the challenge materials available during development.

Because this repository contains custom model/tokenizer code, use trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "l0ulan/chinese-babylm-nanochat-d22-step10000"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="auto",
)
prompt = "春天来了,公园里的花"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=120,
do_sample=True,
temperature=0.8,
top_k=50,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=False))
trust_remote_code=True.Appropriate uses:
Not recommended as-is:
This checkpoint was trained with the karpathy/nanochat codebase and evaluated with the Chinese BabyLM eval pipeline from SiyuanSong2004/chinese-babylm-eval-pipeline.