Guido-1-0.5B-Base

Guido-1-0.5B-Base

A ~553M-parameter math-focused base language model (PiCOFormer, dense decoder-only), pre-trained from scratch on ~77.6B tokens of a math-dominant corpus. Tokenizer: Mathstral-32k. This is a base model — not instruction-tuned, not aligned.

Architecture

• 24 layers · d_model 1280 · 20 heads (head_dim 64) · FFN 3840 · ~553M params
• Channel mixer: SwiGLU (GLU + SiLU)
• Attention: gated multi-head (sparse per-head output gate, gate_input_dim=128), RoPE base 1e6, QK-norm
• Tied input/output embeddings · logit soft-cap 30 · smear-gate
• fp32 master weights, bf16 autocast

Training data — Corpus v5 (QA-dominant + full math, ~77.6B tokens)

Mixture during the stable phase (QA/CoT 0.65 / math-web 0.35):

Data annealing: the final ~12% of training switches to a decay mixture that reinforces the boxed/code answer format (more OpenMathInstruct/TinyGSM/Orca), which keeps the answer-format healthy through warmdown.

Training recipe

• Optimizer: Muon (Polar-Express orthogonalizer, matrix LR 0.01) for 2-D weights; Adam for embeddings (LR 5e-3) and scalars (LR 4e-2).
• Schedule: WSD — warmup 2000 · plateau · linear warmdown 18000 to ~0 (decay-to-zero).
• Batch / compute: 524,288 tokens/step · 148,000 steps · 4 nodes × 4× A100-64GB.

Prompt format (matches pre-training — use as-is, zero-shot)

• MATH: ### MATH\n\nProblem: {problem}\nSolution:
• GSM (code): ### TinyGSM\n\nQuestion: {question}\nAnswer:\n

Results — full benchmarks, post-warmdown

| Benchmark | pass@1 (greedy) |---|---|---|---| | GSM8k (n=1319) | — | 74.0% | | MATH-500 (n=500) | 52.2% |

maj@16 = self-consistency (T=0.6, top_p 0.95, top_k 40). GSM is solved by generating executable Python; MATH answers are \boxed{{}}. MATH pass@1 (greedy) is a slight under-estimate (~16% of greedy completions truncate at 1024 new tokens).

Loading (via Aplos)

from Vathos.guido import load_guido
# this checkpoint uses the SwiGLU channel (separate expand/up projections + SiLU):
model, cfg = load_guido("Paerle/Guido-1-0.5B-Base", channel="torch_swiglu")

Requires the Aplos repo (https://github.com/MarioPaerle/Aplos). The torch_swiglu channel loads the checkpoint 1:1 (no weight fusion) and runs anywhere PyTorch runs. SFT/GRPO examples + docs: examples/guido_{sft,grpo}.py, docs/GUIDO_GRPO.md.

Limitations

Base, math-domain, English-only. Not instruction-tuned / not aligned. GSM is solved via generated Python code — a fraction (~20%) of generated programs fail to execute.