haru_2

<img src="assets/haru.png" alt="Haru character" width="420" />

Haru v2.0 (legacy)

Legacy release / 레거시 모델: Haru v2 is preserved for research and reproducibility. Haru v3 is the latest release. / 최신 버전은 Haru v3입니다.

새로 사용할 때는 학생 non-IT 모델로 동화를 이어 쓰거나, 학생 IT 모델의 apply_chat_template 사용 예제를 참고하세요. 최신 저장소에는 FP32 가중치, MIT 라이선스와 실행 예제가 포함되어 있습니다. IT는 실험 모델이며 이야기의 사실 유지와 새로운 지시 형식에 대한 일반화에는 한계가 있습니다.

See the Haru collection and current GitHub release.

The original v2 architecture, weights and usage are documented below.

Haru is a compact Korean story continuation model built with the custom CFRD causal architecture. It has 16,983,213 parameters and runs at recurrent depth 6 only.

• Source code
• Previous Haru release

Usage

Review the included Python files before enabling remote custom code.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "alice-noa-chan/haru_2"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True)

inputs = tokenizer("작은 마을에 아침이 찾아왔어요.", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    repetition_penalty=1.08,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Model details

• Parameters: 16,983,213
• Context length: 512
• Recurrent depths supervised during training: none; AUX_EXIT_LOSS_WEIGHT is 0.0, so the depth 2 and 4 exits are untrained
• Exported checkpoint step: 32500
• Training tokens seen: 4,259,840,000

Evaluation

Training data attribution

Tiny-Ko-Stories by psymon, licensed under CC BY 4.0. The dataset is not redistributed with this model.

Limitations

• Haru is a continuation model, not an instruction-following assistant.
• Longer generations can repeat ideas or drift between entities.
• The model is not suitable for factual or safety-critical use.
• There is no inference cache yet, so generation recomputes the active context.

License

Haru model weights and included code are released under the MIT License. The training dataset remains under its separate CC BY 4.0 license.

KoBEST

Scored with lm-eval on the five KoBEST tasks. The tasks mix two-way and four-way formats, so the mean has no fixed floor: 0.450 is what a coin flip scores. Read the "vs chance" column.

v1.1 was at or below chance on every task. v2.0 is above chance on four of five, and hellaswag at 0.314 against a 0.25 chance level is the first evidence in this project of a task being learned rather than guessed.

Against other sub-20M Korean models

Every row was scored locally through the same harness and task list rather than quoted from a model card.

These are small margins on a benchmark whose chance mean is 0.450, and all four models score below chance on wic. Sub-20M Korean models are all near the floor; Haru v2.0 is a little further from it than the others tested.

Recurrent depth

v2.0 runs at depth 6 only.

v1.1 degraded gracefully across depths 2, 4 and 6 because it reused three physical cells and trained the intermediate exits with deep supervision. v2.0 unfolds the stack into six independent cells and sets the deep-supervision weight to zero, so the shallow exits receive no training signal. Early exit was a property of cell reuse plus deep supervision, and v2.0 dropped both.

Training

17.0M parameters, 12,000-token BPE tokenizer, 512-token context. Trained on 4.26B tokens drawn from a 2.6B-token Korean corpus that was n-gram decontaminated against all five KoBEST evaluation sets.

Training stopped at 4.26B of a planned 10.4B tokens: two KoBEST measurements 1.5B tokens apart moved the mean by +0.003, inside the harness's own noise, so the remaining budget was not buying measurable accuracy.

Limitations

• Repetition collapse is common after the first sentence or two.
• Entity tracking is unreliable; the model confuses referents in longer spans.
• It reproduces encyclopaedic surface form without the underlying facts.
• Not an instruction model, a factual assistant, or a safety-reviewed product.

Demo Spaces were retired at the owner's request. All model weights and revision history are preserved.

Haru model collection