hellora-olmoe-1b-7b-gsm8k-seed42

HELLoRA OLMoE-1B-7B GSM8K — seed 42

Experimental adapter-only checkpoint from the single-GPU HELLoRA replication at sanitized repository commit 8c2f3870a3d462da430390980e062d99dc5e12d6.

• Source: https://github.com/toolanzyhhh1234/HELLoRA-replication
• Evaluation archive: https://huggingface.co/datasets/toolazyhhh123/hellora-olmoe-gsm8k-evaluations-seed42

Checkpoint

• Base model: allenai/OLMoE-1B-7B-0125
• Base revision: 9b0c1aa87e34a20052389dce1f0cf01da783f654
• Dataset revision: 740312add88f781978c0658806c59bc2815b9866
• Method: HELLoRA, layer-level top-8 hot experts
• LoRA rank/alpha: 32/64
• Targets: attention, routers, selected experts
• Trainable parameters: 47,218,688 (0.6824%)
• Seed: 42
• Weight SHA-256: 07cef4b06c0022db444ef7e21bed6e6b4e9393d55aa3aa26d791cd2cc339ebf3

This is a custom HELLoRA checkpoint, not a standard PEFT checkpoint. The selective expert topology must be reconstructed from hot_experts.json before loading adapter_model.safetensors. Use the loader in the HELLoRA-replication source repository.

Held-out GSM8K result

Under the replication's format-tolerant answer extraction:

• pretrained base: 46.47% (613/1,319);
• HELLoRA: 39.58% (522/1,319).

This is not a parser-neutral base comparison. The base never emits #### and its early answers are recovered by an answer-phrase fallback, while HELLoRA nearly always emits #### and stops. A strict shared marker regex scores the base at 0.00% and HELLoRA at 39.58%; a shared final-number regex scores them at 1.97% and 39.58%, respectively.

The adapter did learn the requested output format: 99.55% of generations used the #### marker and stopped at EOS.

Files

• adapter_model.safetensors: adapter tensors only
• adapter_config.json: topology, revisions, and checksum
• hot_experts.json: selected experts for every MoE layer
• run_config.json: training configuration
• metrics.json: training and checkpoint verification metrics