Downloads · 30 days
93
32% of all-time downloads
nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-base
Llama-3.2-tiny-lm-v2-japanese-500m-base is a text generation model from nagohachi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning followed by continued pre-training in Japanese.
Downloads · 30 days
93
32% of all-time downloads
All-time downloads
294
Public
Parameters
476M
953 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors953 MB · 99%
From the Hugging Face model README
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning followed by continued pre-training in Japanese.
A ~476M-parameter Japanese base model created by width-pruning Llama-3.2-3B down to a 500M-class model, swapping in the Japanese llm-jp/llm-jp-3-440m tokenizer, and recovery-pretraining on ~10B tokens of educational Japanese web text. This is the base model — no instruction tuning, no chat template. See Llama-3.2-tiny-lm-v2-japanese-500m-sft and Llama-3.2-tiny-lm-v2-japanese-500m-dpo for the instruction-tuned / preference-aligned variants.
It is the pruning-based counterpart to the from-scratch tiny-lm-v1-japanese-500m-base; at an equal ~10B-token budget it reaches the from-scratch model's final quality in roughly half the tokens and scores higher on downstream tasks (see Evaluation).
The model uses a custom Llama-style implementation shipped with the repo
(trust_remote_code=True required).
| Parameters | 476M |
| Source model | meta-llama/Llama-3.2-3B |
| Pruning | width: hidden 3072→1024, heads 24→8, KV 8→4 (head_dim 128 kept), 28 layers |
| Context length | 2048 |
| Vocab size | 99,574 |
| Tokenizer | llm-jp/llm-jp-3-440m (embeddings remapped mean-of-pieces from Llama) |
sample_10BT)from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")
inputs = tok("日本の首都は", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))
We evaluated the models using 100 examples from the test split with greedy
decoding. The llm-jp baselines were re-evaluated with the same harness. Base
models use the 4-shot setting; SFT/DPO models and the instruct baseline are
prompted through their chat template. The tiny-lm-v1 rows are the from-scratch
counterparts for reference.
| Model name | average | CR | EL | FA | HE-EN | HE-JA | MR | MT | NLI | QA | RC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.2-tiny-lm-v2-japanese-500m-base | 0.232 | 0.270 | 0.322 | 0.061 | 0.170 | 0.100 | 0.013 | 0.657 | 0.398 | 0.137 | 0.190 |
| Llama-3.2-tiny-lm-v2-japanese-500m-sft | 0.224 | 0.320 | 0.229 | 0.058 | 0.153 | 0.090 | 0.017 | 0.665 | 0.405 | 0.112 | 0.190 |
| Llama-3.2-tiny-lm-v2-japanese-500m-dpo | 0.222 | 0.320 | 0.225 | 0.064 | 0.153 | 0.060 | 0.017 | 0.664 | 0.395 | 0.108 | 0.210 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| tiny-lm-v1-japanese-500m-base | 0.198 | 0.223 | 0.130 | 0.035 | 0.207 | 0.130 | 0.007 | 0.504 | 0.483 | 0.072 | 0.190 |
| tiny-lm-v1-japanese-500m-sft | 0.191 | 0.310 | 0.102 | 0.036 | 0.187 | 0.070 | 0.013 | 0.523 | 0.380 | 0.082 | 0.210 |
| tiny-lm-v1-japanese-500m-dpo | 0.194 | 0.310 | 0.130 | 0.036 | 0.187 | 0.070 | 0.013 | 0.531 | 0.345 | 0.082 | 0.240 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| llm-jp/llm-jp-3-440m | 0.249 | 0.280 | 0.266 | 0.059 | 0.170 | 0.080 | 0.027 | 0.760 | 0.525 | 0.144 | 0.180 |
| llm-jp/llm-jp-3-440m-instruct3 | 0.254 | 0.317 | 0.289 | 0.066 | 0.223 | 0.110 | 0.017 | 0.744 | 0.520 | 0.155 | 0.100 |
This model is a derivative of Meta Llama 3.2 and is distributed under the Llama 3.2 Community License (see also the Acceptable Use Policy). Attribution is provided in the NOTICE file. Built with Llama.