honkhazard-2

<font size=+4 face="monospace">honkhazard-2</font><br><font size=+1 face="monospace" color="#aaa">61.6M (20.97M embed, 16L/8H) | 1.23B seen | 64K vocab</font>

a second experiment to train only on synthetic messages! same config as honkhazard-1 but scaled up. failed but published because why not :)

• parameters: 61.6M (20.97 embed, 20.97 head, 13.11 mlp, 6.55 attn)
• tokens seen: 1.23B
• num_layers: 16
• num_heads: 8
• vocab_size: 65536

<br>

trained on 1x rtx 5090 in 129.6m: <img width=800 src="https://cdn-uploads.huggingface.co/production/uploads/68bef4a6ca845e4b309b7485/RG-vq4kfxYYA6B0nplcTw.png">

<br>

pre-training

pre-trained only on SYNTH messages in the following format:

<|bos|><|user_start|>{{query}}<|user_end|><|assistant_start|><|reasoning_start|>{{synthetic_reasoning}}<|reasoning_end|>{{synthetic_answer}}<|assistant_end|>

post-training

no post-training of any form has been performed on this model

postmortem

definitely more capable than honkhazard-1, it shows some real understanding of prompts but fails to give an answer correctly. it now follows completely the <|reasoning_start>talking to self<|reasoning_end>more human answer<|assistant_end|> naturally. it can answer "What is 2 + 2?" as 4 but gets confused/wrong if other numbers. often gets stuck repeating one token (sometimes temporarily, sometimes forever). likely still parameter limited