HRM-Text-1B-Chat

HRM-Text-1B-Chat — Two-Phase Chat Fine-Tune

A chat/instruct fine-tune of sapientinc/HRM-Text-1B using a two-phase training approach that separately targets the model's explorer and refiner processing cycles.

What is HRM?

HRM (Hierarchical Recurrent Model) processes each token through 8 sequential steps: 2 H-cycles × 3 L-cycles. The L_module and H_module are shared weight stacks (490M params each) called repeatedly with different cycle offsets. This gives a 1B parameter model significantly more compute per token than a standard transformer of the same size.

Training Approach

The base HRM-Text-1B uses torch.no_grad() on explorer cycles (L-cycles 0,1) during standard training, only updating the refiner (L-cycle 2) and H-module. This fine-tune removes that constraint and trains in two phases:

Phase 1 — Explorer Training:

• Trainable: L_module + embeddings (58.5% of params), H_module frozen
• Data: 2,400 balanced examples from UltraChat 200K
• LR: 1e-5, 2 epochs, cosine decay
• Result: Model learns conversational structure and formatting

Phase 2 — Full Training + Identity Grounding:

• Trainable: 100% of parameters, starting from Phase 1 weights
• Data: 2,399 examples (92% fresh UltraChat + 8% synthetic)
  • 60 identity examples, 30 factual corrections, 30 epistemic calibration
  • 24 humor/creative, 24 grounding, 15 structured reasoning
• LR: 2e-5, 3 epochs, cosine decay
• Result: Model gains identity awareness, factual grounding, and conversational polish

Results

Evaluated on a 30-prompt benchmark across 10 categories (formal, technical, casual, creative, silly, adversarial, conversational, instruction, identity, edge cases).

Identity Examples

Token Leak Rate by Category

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "taykso/HRM-Text-1B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype=torch.float32)

prompt = "<|im_start|>user\nWhat is your name?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9, do_sample=True, repetition_penalty=1.1)

response = tokenizer.decode(output[0][inputs['input_ids'].shape[1]:], skip_special_tokens=False)
if "<|im_end|>" in response:
    response = response[:response.index("<|im_end|>")]
print(response)

Observations

• Both training phases exhibit a U-shaped loss curve: loss increases in epoch 1 then recovers. This appears to be a fundamental property of training all HRM cycles simultaneously.
• 8% synthetic data was sufficient to ground identity and factual correction without degrading general chat quality.
• The model runs at ~5 tok/s due to the 8-cycle architecture (equivalent compute to 8 forward passes per token).
• The two-phase approach allows targeted improvement of explorer vs refiner capabilities.

Training Details

• Hardware: AMD Radeon 8060S (Strix Halo APU), ROCm 7.2
• Total time: ~62 hours (Phase 1: ~28h, Phase 2: ~42h)
• Framework: PyTorch + Transformers
• Source patch: Line 525 of modeling_hrm_text.py — removed torch.no_grad() from explorer cycles
• ChatML format: <|im_start|>role\ncontent<|im_end|>

Limitations

• 1B model — limited world knowledge, will hallucinate on specific facts
• Humor understanding is improved but still literal
• Cannot do multilingual translation
• Slow inference (~5 tok/s) due to 8-cycle recurrent architecture
• Trained on English only

License

Apache 2.0 (same as base model)