Downloads · 30 days
107
36% of all-time downloads
nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-sft
Llama-3.2-tiny-lm-v2-japanese-500m-sft is a text generation model from nagohachi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning, continued pre-training, and supervised fine-tuning.
Downloads · 30 days
107
36% of all-time downloads
All-time downloads
294
Public
Parameters
476M
953 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors953 MB · 99%
From the Hugging Face model README
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning, continued pre-training, and supervised fine-tuning.
Instruction-tuned version of Llama-3.2-tiny-lm-v2-japanese-500m-base — a ~476M-parameter Japanese model created by width-pruning Llama-3.2-3B and recovery-pretraining on ~10B tokens. It is the pruning-based counterpart to the from-scratch tiny-lm-v1-japanese-500m-sft.
The model uses a custom Llama-style implementation shipped with the repo
(trust_remote_code=True required) and the llm-jp-3 instruct chat template.
| Parameters | 476M |
| Source model | meta-llama/Llama-3.2-3B |
| Context length | 2048 |
| Vocab size | 99,574 |
| Tokenizer | llm-jp/llm-jp-3-440m |
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-sft"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")
messages = [{"role": "user", "content": "日本の首都はどこですか?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
We evaluated the models using 100 examples from the test split with greedy
decoding. The llm-jp baselines were re-evaluated with the same harness. Base
models use the 4-shot setting; SFT/DPO models and the instruct baseline are
prompted through their chat template. The tiny-lm-v1 rows are the from-scratch
counterparts for reference.
| Model name | average | CR | EL | FA | HE-EN | HE-JA | MR | MT | NLI | QA | RC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.2-tiny-lm-v2-japanese-500m-base | 0.232 | 0.270 | 0.322 | 0.061 | 0.170 | 0.100 | 0.013 | 0.657 | 0.398 | 0.137 | 0.190 |
| Llama-3.2-tiny-lm-v2-japanese-500m-sft | 0.224 | 0.320 | 0.229 | 0.058 | 0.153 | 0.090 | 0.017 | 0.665 | 0.405 | 0.112 | 0.190 |
| Llama-3.2-tiny-lm-v2-japanese-500m-dpo | 0.222 | 0.320 | 0.225 | 0.064 | 0.153 | 0.060 | 0.017 | 0.664 | 0.395 | 0.108 | 0.210 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| tiny-lm-v1-japanese-500m-base | 0.198 | 0.223 | 0.130 | 0.035 | 0.207 | 0.130 | 0.007 | 0.504 | 0.483 | 0.072 | 0.190 |
| tiny-lm-v1-japanese-500m-sft | 0.191 | 0.310 | 0.102 | 0.036 | 0.187 | 0.070 | 0.013 | 0.523 | 0.380 | 0.082 | 0.210 |
| tiny-lm-v1-japanese-500m-dpo | 0.194 | 0.310 | 0.130 | 0.036 | 0.187 | 0.070 | 0.013 | 0.531 | 0.345 | 0.082 | 0.240 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| llm-jp/llm-jp-3-440m | 0.249 | 0.280 | 0.266 | 0.059 | 0.170 | 0.080 | 0.027 | 0.760 | 0.525 | 0.144 | 0.180 |
| llm-jp/llm-jp-3-440m-instruct3 | 0.254 | 0.317 | 0.289 | 0.066 | 0.223 | 0.110 | 0.017 | 0.744 | 0.520 | 0.155 | 0.100 |
This model is a derivative of Meta Llama 3.2 and is distributed under the Llama 3.2 Community License (see also the Acceptable Use Policy). Attribution is provided in the NOTICE file. Built with Llama.