HRM-Tiny-JA-500k-2x2-RC1

HRM Tiny JA 500k 2x2 RC1 / 非公式 HRM-Text 小型日本語 RC1

Important / 重要
This is an unofficial experimental model. It is not released, endorsed, or certified by Sapient Intelligence, Hugging Face, or LLM-jp.
本モデルは 非公式の実験モデル です。Sapient Intelligence、Hugging Face、LLM-jp による公式モデル、公式派生、認定モデルではありません。

This repository draft is for the release candidate selected from:

./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500

Suggested public-facing name:

hrm-tiny-ja-500k-2x2-rc1

This checkpoint is intended as a small Japanese instruction / explanation generation experiment based on the HF HRM-Text implementation. It is not a general-purpose chat model and should not be used as a factual authority.

---

1. Model summary / モデル概要

日本語説明生成の検証用途を主目的とした小型モデルです。RC1では、2 epoch 学習の最終checkpointではなく、1 epoch相当の checkpoint-62500 を採用しています。1.5 epoch候補はfirst-token指標がやや良い一方で、本文がプロンプトから飛びやすかったため採用していません。

---

2. Prompt format / プロンプト形式

Training and evaluation used the following plain instruction format.

<s>以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。

### 指示:
{instruction}

### 応答:

The model was trained with PrefixLM-style masks. In local inference, prompt tokens should be assigned token_type_ids=1, and generated response tokens should be assigned token_type_ids=0.

---

3. Quick start / 推論サンプル

Install

pip install -U "git+https://github.com/huggingface/transformers.git" torch safetensors

HRM-Text support may require a recent Transformers version. The official HRM-Text documentation shows the general pattern of using AutoTokenizer and AutoModelForCausalLM.generate(...) for HRM-Text style models, while noting that the official HRM-Text-1B base model does not ship a stable chat template. For this RC, use the plain prompt above and the bundled inference script if PrefixLM token_type_ids handling is needed.

Recommended local script

python scripts/inference_hrm_prefixlm.py \
  --model_dir ./release/hrm_tiny_ja_500k_2x2_rc1 \
  --instruction "自然言語処理とは何か" \
  --max_new_tokens 128 \
  --temperature 0.65 \
  --top_p 0.9 \
  --top_k 50 \
  --repetition_penalty 1.2 \
  --no_repeat_ngram_size 4

Python API example

from pathlib import Path
from scripts.inference_hrm_prefixlm import load_model_and_tokenizer, generate_text

model_dir = Path("./release/hrm_tiny_ja_500k_2x2_rc1")
model, tokenizer, device = load_model_and_tokenizer(model_dir)

text = generate_text(
    model=model,
    tokenizer=tokenizer,
    device=device,
    instruction="人工知能と機械学習の違いを説明してください",
    max_new_tokens=128,
    temperature=0.65,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.2,
    no_repeat_ngram_size=4,
)
print(text)

---

4. Training data / 学習データ

The RC1 dataset is:

./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean

Target composition after filtering and source sampling:

Validation splits:

Key filters:

• Require Japanese text in instruction and response.
• Drop numeric-only, symbol-heavy, refusal/meta, and apology-style responses.
• Drop many list-like or numbered responses.
• Drop repeated phrase / repeated n-gram responses.
• Drop long context extraction tasks and prompt-heavy examples.
• Token constraints: default min_response_tokens=24, max_response_tokens=320, max_prompt_tokens=448, max_total_tokens=896.
• Banned first response tokens include Output, Solution, first, second, はい, いいえ, 以下, list markers, and numeric starts 0 through 20.

See cards/DATASET_CARD_DRAFT.md for a fuller dataset description.

---

5. Training setup / 学習設定

Selected RC1 run:

CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes 1 train_hrm_tiny_sft.py \
  --dataset_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
  --eval_split_name validation_core \
  --output_dir ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
  --num_layers_per_stack 2 \
  --max_epochs 2 \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 1 \
  --learning_rate 2e-4 \
  --weight_decay 0.05 \
  --warmup_ratio 0.03 \
  --loss_normalization active_tokens \
  --first_response_token_weight 4 \
  --first_response_token_target first_non_space \
  --embedding_init_mode raw \
  --lm_head_init_mode random \
  --eval_strategy epoch \
  --save_strategy epoch \
  --save_total_limit 4

Training metrics for the full 2-epoch run:

Selected checkpoint:

checkpoint-62500

---

6. Evaluation summary / 評価要約

Primary evaluation on release_ja_500k_clean / validation_core, first 256 examples:

Legacy comparison on old mixfix_20k_ja4x / validation_ja_dolly, first 256 examples:

Known behavior:

• The model often selects a relevant first token for definition/explanation prompts.
• It can produce Japanese explanation-like text.
• It is still weak at factual accuracy, multi-sentence coherence, and causal explanations.
• It can hallucinate details, confuse related concepts, or drift into generic explanations.
• It is not suitable for safety-critical or factual-critical use.

See cards/EVAL_SUMMARY.md for selected logs and comparison notes.

To prepate this dataset, you can use prepare_release_ja_dataset_500k.py.

python prepare_release_ja_dataset_500k.py \
  --output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
  --num_proc 8 \
  --overwrite

より厳しめに長文contextや長い応答を落とす場合はこちらです。

python prepare_release_ja_dataset_500k.py \
  --output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean_strict \
  --max_context_chars 700 \
  --max_total_tokens 768 \
  --max_response_tokens 256 \
  --num_proc 8 \
  --overwrite

---

7. Example generations / 生成例

Generation parameters used for sanity checks:

temperature=0.65
top_p=0.9
top_k=50
repetition_penalty=1.15 or 1.2
no_repeat_ngram_size=3 or 4
max_new_tokens=128

Example prompt:

自然言語処理とは何か

Observed RC1-style output excerpt:

自然言語処理（NLP）は、人間が言葉を理解し、生成するための技術で、...

The sample is not guaranteed to be factually accurate. Use the model as an experimental small HRM-Text checkpoint, not as a factual assistant.

---

8. Limitations / 制限事項

• 非公式・研究用の小型モデルです。
• 事実誤認、幻覚、定義の混線、文章の途中崩れがあります。
• 日本語説明生成に寄せていますが、チャットモデルとして安定していません。
• 長文コンテキスト、厳密な数学、コード生成、翻訳、要約、抽出には向いていません。
• 医療・法律・金融・安全に関わる判断には使わないでください。
• Dataset and upstream license compatibility must be checked before public redistribution.

---

9. Release checklist / 公開前チェック

Before public upload:

1. Copy checkpoint-62500 into a clean release directory.
2. Copy tokenizer files from the training output directory.
3. Run scripts/evaluate_rc1.sh and save logs under eval_logs/.
4. Confirm source dataset licenses and redistribution constraints.
5. Update model repo ID, author, date, and license field.
6. Add clear disclaimer: unofficial, experimental, no warranty.

Suggested packaging command:

bash scripts/package_rc1_checkpoint.sh \
  ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500 \
  ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
  ./release/hrm_tiny_ja_500k_2x2_rc1