LS2.5-108M-A17M-Coder-GGUF

LS2.5: Efficientmaxxing at its finest

LS2.5 Expands upon LS2 by introducing a new 16k vocab tokenizer instead of the 8K vocab. This increases efficiency (same dataset is ~12% less tokens than 8k variant) but active parameters also increases (~3M increase).

A new 5.7B pretrain dataset was created with the following sources:

• Ultrafineweb
• Github-python
• Cosmopedia-v2
• ufw-l3
• finemath
• hermes-2.5
• ultrachat
• toolace
• hermes-fc
• glaive-v2
• heremes-single
• heremes-gl5k
• apigen-mt

LS2.5-108M-A17M-Coder

This repo contains the Coder sft variant of LS2.5-108M-A17M-base. It is additionally fine tuned on ~1B tokens of Agentic coding sft.

A 108M MoE model with top-2 experts routing, entirely trained on ~5.7B tokens with a 5060ti over ~26h.

LS2.5-108M-A17M retains 94% of benchmark accuracy of both models flagship models while being 2.4 times smaller.

Internal benchmark:

118 Multiple choice questions and 24 open prompts was benched on LS2.5-108M-A17M-Base and the two sft variants. The multiple choice questions covered facts, common sense, following instructions, picking-on-the-topic reply, staying consistent with the conversation and simple arithmetic. Random guessing scores 25%.

The model responds to the open prompts with 4 samples + 1 greedy reply. They are checked for the following:

• ends by itself, or hit the 128 token cap
• if it mentions the topic
• if it loops
• how varied wording is
• how the reply depends on the prompt vs other topics

Overall, the fine tuned models don't gain any knowledge (benchmarks pretty much unchanged) but rather their output format and reply length.

Examples:

"chat" (temp 0.2)

you> hi
bot> You're welcome! If you have any other questions, feel free to ask.
      [16 tok, 1454.5 tok/s, ctx 26/4096]

you> What is the capital of France?
bot> The capital of France is Pope Francis.
      [10 tok, 1250.0 tok/s, ctx 26/4096]

you> which colour is the sky?\nA:blue\nB:green\nC:purple\nD:pink
bot> The sky is a beautiful and vibrant colors that make it look like a blue. The sky has a bright, warm glow, while the green light creates an airy atmosphere. It's also a natural beauty place to enjoy in your life.
      [48 tok, 1090.9 tok/s, ctx 85/4096]

Usage: To run this model, download the latest tinylm from github and build from source. Currently working on windows prebuilts. Then run tinylm chat LS2.5-108M-A17M-Coder-Q4_K_M.gguf 512 temp. Try temp 0.2 for general chat and 0.7 for nonsensical yap.