open-reason-xl

Open Reason XL (CPU)

GPT-2-style causal LM trained from scratch on the Open Reason SFT split. Exact parameter count: 443,719,680. This is not a 1B model and is not theworker02/open-reason-1b.

Training facts

• Parameters: 443,719,680 (sum(p.numel() for p in model.parameters()))
• Architecture: n_layer=22, n_embd=1280, n_head=20, vocab=8192, seq=256
• Steps: 120 (batch 1, gradient accumulation 2, gradient checkpointing)
• Device: host CPU (AMD Ryzen 9 9950X, 32 threads)
• torch: 2.12.0+cpu; cuda_available=False; Docker not installed
• NVIDIA CUDA was not used. AMD GPU / ROCm / DirectML were not used
• Dataset: theworker02/open-reason pipeline 1.4.0
• SFT rows: 3175 (data/release/all.jsonl)
• Final training NLL: 5.8116
• License: Apache-2.0
• No held-out exact-match / coding / math benchmark scores are claimed

Related checkpoints (none of these is a 1B model):

• Dataset: https://huggingface.co/datasets/theworker02/open-reason
• Small: https://huggingface.co/theworker02/open-reason-small
• Medium: https://huggingface.co/theworker02/open-reason-medium
• Large: https://huggingface.co/theworker02/open-reason-large
• XL: https://huggingface.co/theworker02/open-reason-xl

No Reddit sources. Project license Apache-2.0.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("theworker02/open-reason-xl")
model = AutoModelForCausalLM.from_pretrained("theworker02/open-reason-xl")