open-reason-large

Open Reason large (CPU)

This is a large GPT-2-style causal LM trained from scratch on the Open Reason SFT split. It is larger than theworker02/open-reason-medium (13,867,008 parameters) and is not a 1B model and is not theworker02/open-reason-1b.

Weights live on this Hub repo. They are not stored in the GitHub git tree.

Training facts

• Parameters: 91,544,064
• Architecture: GPT-2-style from scratch; n_layer=12, n_embd=768, n_head=12, vocab_size=8192, max_seq_len=256
• Steps: 400 (batch size 2)
• Final training loss: 5.7361 (next-token NLL on training batches; not a benchmark score)
• SFT rows: 3,175 from data/release/all.jsonl
• Dataset: theworker02/open-reason pipeline 1.4.0
• License: Apache-2.0
• Hardware: AMD Ryzen 9 9950X host CPU; torch 2.12.0+cpu; torch.cuda.is_available()=False; Docker not installed and not used; nvidia-smi not present. NVIDIA CUDA was not used. AMD GPU / ROCm / DirectML were not used.

Related

• Dataset: https://huggingface.co/datasets/theworker02/open-reason
• Small (~1.3M): https://huggingface.co/theworker02/open-reason-small
• Medium (13,867,008): https://huggingface.co/theworker02/open-reason-medium
• Large (this repo, 91,544,064): https://huggingface.co/theworker02/open-reason-large
• GitHub: https://github.com/theworker02/open-reason

No Reddit sources.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("theworker02/open-reason-large")
model = AutoModelForCausalLM.from_pretrained("theworker02/open-reason-large")