Rambley-150M-RealBase

150M model with OLMo-3 style arch (with 2:1 SWA with a 512 token window), pretrained on ~150B tokens of Dolma 3. Compute sponsored by lium.io, thank you <3!

Motivation

I swear to god, if I see one more of these random Huggingface posts made by a 5 year old using Claude to pretrain a 150M with some homebrewed datamix, I'm going to lose it. Why not make my own? :) (it was also a fun learning experience so oh well)

Ablations

This cost way too much so I didn't have time for many ablations, but the main thing I tried was XSA attention but I found it to not converge as quickly so we just use regular attention

Benchmarks

While not SOTA by any means, for not having mixed in any benchmaxxing/domain-specific data in the actual pretraining mix, it's not too bad

Code / Logs

• Pretraining/midtraining codebase: https://code.allura.moe/fizz/OLMo-core
• Wandb pretraining logs
• Wandb midtraining logs

<details> <summary>Other random benchmarks</summary>

BananaMind Base Bench 1.1
Overall Elo: 1006
Accuracy: 183/350 (52.29%)
Weighted accuracy: 48.76%
language_completion: Elo 1331 | 47/50 (94.00%) | weighted 93.31%
commonsense: Elo 1043 | 33/50 (66.00%) | weighted 62.15%
world_knowledge: Elo 1031 | 31/50 (62.00%) | weighted 60.50%
context_tracking: Elo 859 | 16/50 (32.00%) | weighted 31.53%
quantitative: Elo 822 | 11/50 (22.00%) | weighted 21.77%
logical_reasoning: Elo 1043 | 23/50 (46.00%) | weighted 42.32%
code_completion: Elo 1045 | 22/50 (44.00%) | weighted 43.31%

Loaded 1000 ArithMark 3.0 examples from /root/arith/arithmark-3.jsonl
ArithMark 3.0 dataset SHA-256: bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435
Using device: cuda

====================================================================
  Loading allura-forge/Rambley-150M-RealBase...
====================================================================
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 178/178 [00:00<00:00, 5401.16it/s]
  149,935,616 parameters (torch.bfloat16)
  arithmark-3: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:02<00:00, 15.98it/s]
  arithmark-3: raw 37.20% (372/1000)  normalized 37.20% (372/1000)
  speed: 499.2 examples/s  (tokenize 0.44s, evaluate 2.00s)

  Category                                                             N       Raw   Normalized
  ----------------------------------------------------------------------------------------------
  elementary_school_math_continuation::addition::grades_1_2::easy    128    25.00%       25.00%
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    22.73%       22.73%
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    25.00%       25.00%
  elementary_school_math_continuation::data::grades_2_3::easy         43    20.93%       20.93%
  elementary_school_math_continuation::division::grades_3_4::medium    54    33.33%       33.33%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    22.00%       22.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    73.08%       73.08%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    60.00%       60.00%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    30.26%       30.26%
  elementary_school_math_continuation::money::grades_3_4::medium      64    28.12%       28.12%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    63.51%       63.51%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    32.08%       32.08%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    23.08%       23.08%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    23.91%       23.91%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    31.58%       31.58%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    34.38%       34.38%

====================================================================
  allura-forge/Rambley-150M-RealBase (149,935,616 params) RESULTS
====================================================================
  Raw continuation accuracy        37.20%
  Length-normalized accuracy       37.20%
  Primary (acc_norm)           37.20%
====================================================================

</details>