TinyGPT-medium-large

TinyGPT-ML-84M (TGPT-XL) An 85M parameter language model trained entirely from scratch — no pretrained base, no borrowed weights, no fine-tuning on top of someone else's model. Architecture Modern Llama-style stack: RoPE, RMSNorm, SwiGLU ChatML format with a dedicated stop token Trained on 8B tokens (FineWeb-Edu + SmolTalk), heavily overtrained past Chinchilla-optimal for maximum capability-per-parameter Benchmarks Evaluated head-to-head against Supra-1.5-50M-Instruct across 5 categories (42 questions, fair decode settings: temp 0.3, top-p 0.9, repetition penalty 1.15). Category TGPT-XL Supra-50M Knowledge 50% 40% Retention 83% 50% Reasoning 37% 12% Advice 100% 38% Overall 62% 36% TGPT-XL wins or ties every single category, with its biggest edges in multi-turn retention (holds context across turns) and open-ended advice (structured, actionable responses). Supra tends to deflect on emotional/open-ended prompts. Highlights Clean multi-turn context recall Respects length and format constraints Reliable stop behavior Runs on-device (Pixel 6a) via GGUF