Atom-350M-GGUF

Atom-350M GGUF

GGUF quantizations of TinyModels/Atom-350M, for use with llama.cpp and compatible runtimes.

About

Quantizations were produced by using llama.cpp build b10090, via convert_hf_to_gguf.py followed by llama-quantize. Q8_0 and Q6_K are static (non-imatrix) quantizations; Q5_K_M and Q4_K_M use an importance matrix computed with llama-imatrix over 200 chunks of wikitext-2 (wiki.train.raw).

Files

Note on K-quants: this model's tensor dimensions (960, 320, 2560) are not evenly divisible by the 256-element superblock size K-quants require. As a result, 192 of 290 tensors in the Q6_K file fall back to Q8_0 quantization, which is why Q6_K is barely smaller than Q8_0. Expect similar (lesser) fallback behavior in the upcoming Q5_K_M/Q4_K_M builds.

Quantization Comparison

Perplexity and KL-divergence measured on wikitext-2 (wiki.test.raw), 20 chunks, n_ctx=512, using llama-perplexity --kl-divergence against saved F16 reference logits. Tokens/sec measured with llama-bench (pp512/tg128, 4 threads) on Apple Macbook Air M2.

Perplexity & speed

KL-divergence vs F16

Mean PPL can mask token-level disagreement — two quants can tie on PPL while differing in the full predicted distribution. KLD/Δp are measured per-token; "Same top token" is the share of tokens where the quant agrees with F16 on the single most-likely next token.

Notably, Q5_K_M and Q6_K nearly tie on mean PPL (17.63 vs 17.64 above) but not on KLD — Q6_K is ~2.6x lower and has better top-token agreement, so it's the better pick when distributional fidelity matters more than the near-tie in PPL suggests.

Q8_0/Q6_K are static quantizations; Q5_K_M/Q4_K_M use an imatrix computed from 200 chunks of wikitext-2 wiki.train.raw. The same tensor-shape fallback noted above applies to all four.

Which Quant Should I Choose?

• Q4_K_M — default pick. Smallest, fastest generation, +2.1% PPL vs F16. Highest KLD (0.028 mean); skip if you need tight distributional fidelity.
• Q5_K_M — 275MB middle ground if Q4_K_M's KLD is too high, but Q6_K generally wins.
• Q6_K — best top-token agreement short of Q8_0 (96.2%), though the tensor-shape fallback puts it near Q8_0's size. Worth the ~75MB over Q5_K_M when accuracy matters.
• Q8_0 — safest, closest to FP16 (97.9% top-token, lowest KLD) if you have ~400MB to spare.
• F16 — reference baseline, or if you'll re-quantize/fine-tune.

How to Run

llama.cpp

llama-cli -m Atom-350M-Q8_0.gguf -p "prompt" -n 128

LM Studio

Search "Atom-350M GGUF", download your preferred quant, load it.

Ollama

ollama run hf.co/syaffers/Atom-350M-GGUF:Q8_0

text-generation-webui

Drop the .gguf file into your models/ folder and select the llama.cpp (GGUF) loader.

Prompt Format

ChatML:

<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant