NanoFlare-v1

NanoFlare is my newest model, based on the recently released Qwen 3.8 27B. It is designed to run completely on GPU if you have 8GB of VRAM, otherwise it can be run on CPU if you have at least 10GB or more of system memory (this will be slow). I have applied a non-uniform quantization across the weights, allocating more precision to the more sensitive parts of the model to preserve as much quality as possible.

v1.c is the latest version (recommended)

Revision C brings more improvements to the quality of output, and coherence in the model's reasoning process. It's been squeezed down to a reduced size so that a small kv cache can be fit onto the GPU as well.

If you are running NanoFlare on an 8GB GPU, you can have up to around an 8k context on GPU (tested with k:q5_0, v:iq4_nl), but if you want it higher you will have to offload the kv cache to CPU.

The MTP draft predictor is stripped out of regular NanoFlare to save space, but there is also a build that includes it, look for "--with-mtp" in the file name. Note, I can not get this version to fit fully onto my own 8GB card, so to use the MTP included version you may need a 10GB GPU or bigger.

Relative perplexity scores (all tested with wikitext2, lower is better):

The model seems to run best at temperature: 0.95 + top-k: 20.

Note this model has been hyper-compressed, so it's not perfect. It may flake out on you from time to time, but it is mostly fairly stable. Taking quantization to the extreme levels I'm using comes at a cost. The LLM will sometimes not complete its thought process. If you find yourself in this situation, try the prompt again. It often works correctly on the second try when this happens. It seems to be pretty good at most general tasks, and is good at coding. It has forgotten some facts and figures from its training data through the heavy quantization, so I'd recommend giving it web search access or double checking anything its telling you from recall.

From subjective, manual testing v1.c seems to be the most stable and usable version so far.

MMLU Pro (limited) Results*

*note only a partial set was ran, the first 20 questions in each of the 14 categories

Tests were ran at a 5 shot with an 8k context, using NanoFlare v1.c. Top-k: 40, Temperature: 0.80 (llama.cpp defaults)

So NanoFlare should give you fairly decent quality, but microFlare scored a 0.73 and will give you more stable output.

For more details, see my announcment blog post.