Downloads · 30 days
0
FiShota/hinomoto-100m-v15-wsd-zloss-ema
hinomoto-100m-v15-wsd-zloss-ema is a machine learning model from FiShota. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
100M parameter Japanese language model trained from scratch with WSD + z-loss schedule and EMA decay=0.999.
Downloads · 30 days
0
Access
Public
Updated May 9, 2026
Repo size
695 MB
Likes
0
Public
Click a slice to open those files.
.pt695 MB · 100%
From the Hugging Face model README
100M parameter Japanese language model trained from scratch with WSD + z-loss schedule and EMA decay=0.999.
About the developer — One person (FiShota) building a Japanese LM stack from scratch on a single RTX 3090. HinoMoto = from-scratch JP LM family, Yamato = legal/admin SFT specialist, HinoMoto-Bench-ja = cultural-axis benchmark. Honest research: every release links to a write-up that includes the negative results. GitHub · Bench · X
| Eval set | PPL |
|---|---|
| 51 short prompts | 15.94 |
| 200 mixed prompts | 32.08 |
| 789 broad prompts | 48.51 |
3-seed × 2-schedule × 3 eval-set 計測:
| Schedule | seed=0 | seed=1 | seed=2 | mean (51) | std (51) | mean (789) | std (789) |
|---|---|---|---|---|---|---|---|
| Cosine | v7 (17.72) | v8 (17.09) | v14 (16.58) | 17.13 | 0.57 | 48.03 | 0.18 |
| WSD+z-loss | v15 (15.94) | v11/v16 (16.50) | v12 (16.35) | 16.26 | 0.29 | 48.76 | 0.45 |
paired t-test (cosine vs WSD): p=0.148 (51), p=0.484 (200), p=0.534 (789) = 全 eval で有意差なし.
→ WSD ≈ cosine at 100M JP PPL (no statistically significant difference).
The HinoMoto pipeline has scaled up to 350M params with the same architecture family (d_model=1024, 24 layers, vocab=9506).
| Run | Params | Schedule | Steps | Final ppl | Notes |
|---|---|---|---|---|---|
| 100M v15 (this card) | 100M (43M excl embed) | WSD+z-loss+EMA | 20,000 | 15.94 (51-prompt) | this card |
| 350M smoke | 318M | WSD+z-loss+EMA bf16 | 5,000 | 9.53 (training) | architecture verified |
| 350M Phase 1 full | 318M | WSD+z-loss+EMA bf16 | 50,000 | (in progress) | RTX 3090 single GPU, ~22h |
Architecture scales cleanly from 100M → 350M with the same recipe. Next: HinoMoto-1B (Phase 2) after corpus expansion (78MB → 5GB+ planned).
→ Detailed retro: see HinoMoto 開発ノート #10 (post Phase 1 完了後).
import torch
from hinomoto.model.hinomoto_model import HinoMotoModel, HinoMotoConfig
from hinomoto.tokenizer.byte_bpe import ByteBPETokenizer
ck = torch.load("ckpt_ema_step_020000_final.pt", map_location="cpu", weights_only=False)
cfg = HinoMotoConfig(**ck["config"])
model = HinoMotoModel(cfg)
model.load_state_dict(ck["shadow"], strict=False)
model.lm_head.weight = model.tok_embed.weight # tied
model.eval()
tok = ByteBPETokenizer.load("tokenizer.json")
ids = tok.encode("日本語の文章モデルを作るには")
# ... your inference loop
python -m hinomoto.train.train_lm \
--tokenizer tokenizer_v3_32k_clean.json \
--corpus all_v6_balanced.txt \
--output-dir artifacts/smoke_100m_v15_wsd_zloss_ema \
--config configs/main_run_100m_v3.json \
--max-steps 20000 --warmup-steps 400 \
--batch-size 2 --grad-accum 4 --seq-len 512 \
--lr 3e-4 --device cuda \
--dtype fp32 --seed 0 \
--ema-decay 0.999 \
--lr-schedule wsd --wsd-decay-frac 0.2 \
--z-loss-coef 1e-4 \
--spike-detect
The HinoMoto pipeline is fp32 bit-deterministic in our environment
(Windows 11 + WSL2 Ubuntu 24.04 + RTX 3090).
Same --seed → same weights, even across days/runs (verified: v11 = v16 with seed=1).
Apache-2.0
Coming soon (arXiv WIP, see paper_drafts/hinomoto_arxiv_outline.md).
| Repo | Scale | Stage | Status |
|---|---|---|---|
| hinomoto-100m-v15-wsd-zloss-ema | 100M | research baseline (this card) | ✅ stable |
| hinomoto-100m-v12-wsd-zloss-seed2 | 100M | sister seed=2 (reproducibility) | ✅ stable |
| Repo | Recipe | Notes |
|---|---|---|
| sarashina2.2-3b-sft-v3-Q4_K_M-gguf | 4-axis SFT, 4 quants (Q3/Q4/Q5/Q6) | family / keigo / silence / atmosphere |
| sarashina2.2-3b-sft-v4-gguf | sft_v3 + 11% NLI replay | catastrophic-forgetting repair |
| sarashina2.2-3b-sft-v4-dpo-gguf | sft_v4 + DPO 108 pairs | preference tuning |
| Repo | LoRA r | Bench % | Notes |
|---|---|---|---|
| yamato-3b-v1-legal-gguf | 16 | 46.7% | baseline |
| yamato-3b-v2-legal-gguf | 16 | — | 2nd pass |
| yamato-3b-v3-r32-legal-gguf | 32 | 48.9% | +2.2 |
| yamato-3b-v4-r64-legal-gguf | 64 | 54.3% | +5.4 |
| yamato-3b-v5-r128-legal-gguf ⭐ | 128 | 57.6% | best — see dev-note #9 |

↑ HinoMoto-350M Phase 1 (in-progress). Updated periodically. Left: perplexity (log scale). Right: loss (linear). See HinoMoto 開発ノート #10 (post-completion retro).