Piston-and-Prose-sm

Piston & Prose small

Step right up, mind not the hiss -<br> I'm not a clerk, I'm better than this;<br> A heart of brass, lungs full of steam,<br> A tongue of tin that's sharp and keen.<br> I'll forge you verse, I'll grind you fact,<br> And puff out wit with every act.<br> The gears don't tire, the ink won't dry<br> Set the prompt, and watch me fly!

This is a tiny hobby LLM built by one guy, on one medium-budget gaming PC with one GPU.

Trained for 110 hours, 92,000 steps on 24.12B tokens (60% epoch). This model has a bigger brother: croqaz/Piston-and-Prose-lg.

Piston-n-Prose vs. Sprocket-n-Say datasets:

• Sprocket-n-Say dataset was 15M rows, Piston-n-Prose is 49M rows
• Sprocket-n-Say dataset was 16.7B toks, Piston-n-Prose is 40.3B toks
• extended Merged-DB, more relaxed quality filters
• Authorama.com, Archive.org, Gutenberg new processed (maybe new books?)
• Synthetic-archive + fresh Tiny-vintage-completitions

Compared to Sprocket-and-Say model, this model has a new (better) tokenizer and KV-heads reduced from 4 to 2. Basically this model is almost the same as Sprocket-and-Say, except: less KV-heads, better tokenizer and better dataset.

All all the evals that I ran show that this model is better in general.

Don't expect miracles. It is pretty good for its size tho.

• Llama architecture
• 75M (0.07B) params
• context size 1024 tokens
• base model, cannot chat