Downloads · 30 days
375
72% of all-time downloads
nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-dpo
Llama-3.2-tiny-lm-v2-japanese-500m-dpo is a text generation model from nagohachi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.2.
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning, continued pre-training, supervised fine-tuning, and DPO.
Downloads · 30 days
375
72% of all-time downloads
All-time downloads
520
Public
Parameters
476M
953 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors953 MB · 99%
From the Hugging Face model README
Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning, continued pre-training, supervised fine-tuning, and DPO.
DPO-aligned version of Llama-3.2-tiny-lm-v2-japanese-500m-sft — a ~476M-parameter Japanese model created by width-pruning Llama-3.2-3B and recovery-pretraining on ~10B tokens (base model). It is the pruning-based counterpart to the from-scratch tiny-lm-v1-japanese-500m-dpo.
The model uses a custom Llama-style implementation shipped with the repo
(trust_remote_code=True required) and the llm-jp-3 instruct chat template.
| Parameters | 476M |
| Source model | meta-llama/Llama-3.2-3B |
| Context length | 2048 |
| Vocab size | 99,574 |
| Tokenizer | llm-jp/llm-jp-3-440m |
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nagohachi/Llama-3.2-tiny-lm-v2-japanese-500m-dpo"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")
messages = [{"role": "user", "content": "日本の首都はどこですか?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
We evaluated the models with llm-jp-eval v1.4.1 using 100 examples
from the dev split of each dataset, 4-shot prompts, and greedy decoding. All models, including SFT/DPO
models, receive the same prompt without a chat template. The llm-jp baselines are the scores reported in
the LLM-jp blog, which also used llm-jp-eval
v1.4.1 on the dev split. The average is the mean of the 10 categories (code generation is excluded). The tiny-lm-v1 rows are the from-scratch counterparts for reference.
| Model name | average | EL | FA | HE | MC | MR | MT | NLI | QA | RC | SUM |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.2-tiny-lm-v2-japanese-500m-base | 0.277 | 0.277 | 0.059 | 0.230 | 0.303 | 0.050 | 0.571 | 0.464 | 0.332 | 0.431 | 0.058 |
| Llama-3.2-tiny-lm-v2-japanese-500m-sft | 0.249 | 0.240 | 0.059 | 0.195 | 0.317 | 0.030 | 0.595 | 0.342 | 0.245 | 0.364 | 0.107 |
| Llama-3.2-tiny-lm-v2-japanese-500m-dpo | 0.249 | 0.220 | 0.063 | 0.195 | 0.307 | 0.030 | 0.595 | 0.340 | 0.242 | 0.391 | 0.105 |
| tiny-lm-v1-japanese-500m-base | 0.210 | 0.075 | 0.035 | 0.215 | 0.300 | 0.010 | 0.451 | 0.528 | 0.229 | 0.208 | 0.051 |
| tiny-lm-v1-japanese-500m-sft | 0.221 | 0.129 | 0.036 | 0.235 | 0.300 | 0.010 | 0.472 | 0.428 | 0.222 | 0.278 | 0.099 |
| tiny-lm-v1-japanese-500m-dpo | 0.220 | 0.137 | 0.035 | 0.235 | 0.293 | 0.010 | 0.476 | 0.412 | 0.232 | 0.268 | 0.102 |
| llm-jp/llm-jp-3-440m | 0.264 | 0.223 | 0.057 | 0.285 | 0.320 | 0.050 | 0.620 | 0.508 | 0.277 | 0.267 | 0.029 |
| llm-jp/llm-jp-3-440m-instruct3 | 0.274 | 0.296 | 0.060 | 0.240 | 0.313 | 0.050 | 0.646 | 0.444 | 0.229 | 0.384 | 0.073 |
This model is a derivative of Meta Llama 3.2 and is distributed under the Llama 3.2 Community License (see also the Acceptable Use Policy). Attribution is provided in the NOTICE file. Built with Llama.