tiny-lm-japanese-500m-base-v1

tiny-lm-japanese-500m-base-v1

A ~500M-parameter Japanese language model pretrained from scratch on ~10B tokens of educational Japanese web text. This is the base model — no instruction tuning and no chat template. See tiny-lm-japanese-500m-sft-v1 and tiny-lm-japanese-500m-dpo-v1 for the instruction-tuned / preference-aligned variants.

The model uses a custom Llama-style implementation shipped with the repo (trust_remote_code=True required).

Model details

Training

• Data: hotchpotch/fineweb-2-edu-japanese (sample_10BT)
• Schedule: 10B tokens

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nagohachi/tiny-lm-japanese-500m-base-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")

inputs = tok("日本の首都は", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation

llm-jp-eval (v2.1.5)

We evaluated the models using 100 examples from the test split with greedy decoding. The llm-jp baselines were re-evaluated with the same harness. Base models use the 4-shot setting; SFT/DPO models and the instruct baseline are prompted through their chat template.