Helios-Nova-306M-Instruct-GGUF

<p align="center"> <img src="helios_nova_banner.svg" alt="Helios Nova 306M-Instruct GGUF" width="100%"/> </p>

Helios Nova 306M-Instruct — GGUF

GGUF quantizations of Helios-Nova-306M-Instruct for local inference with llama.cpp and llama-server, on any backend: NVIDIA (CUDA), Apple Silicon (Metal), AMD/Intel (Vulkan), or CPU.

Helios Nova is a 306M-parameter dense model pre-trained from scratch on 50B tokens of FineWeb-Edu for under USD 190, then instruction-tuned on smol-smoltalk. It is built from standard components (Grouped-Query Attention, per-head QK-Norm applied before RoPE, SwiGLU, RMSNorm pre-norm, tied embeddings), so it maps one-to-one onto an architecture llama.cpp already implements and converts to GGUF with no approximation — the F16 GGUF reproduces the original PyTorch logits bit-for-bit.

The easiest way to run it is the one-file chat client helios_chat.py on GitHub: it launches llama-server, downloads these weights automatically, and applies the correct prompt format and stop settings.

Files

¹ Perplexity from llama-perplexity on out-of-domain literary text (ctx 512). The absolute value is high only because literary fiction is out of domain for a small, education-trained model; what matters is the small relative degradation. SHA-256 checksums are in SHA256SUMS.txt.

Quick start

Recommended: the helios_chat.py client

Install llama.cpp once — on macOS brew install llama.cpp; otherwise download a release for your backend (CUDA / Vulkan / Metal / CPU) from the releases page. Then:

git clone https://github.com/rafaelespinosamena/Helios-Nova-306M-Instruct-GGUF.git
cd Helios-Nova-306M-Instruct-GGUF
pip install huggingface_hub          # optional, for cached downloads

python helios_chat.py                # Q8_0 (default) — weights auto-download
python helios_chat.py --model q4     # Q4_K_M (smaller / faster)
python helios_chat.py --ngl 0        # force CPU

The client locates llama-server on the PATH automatically (or use --server / --bin-dir) and caches the GGUF after the first download.

Using llama.cpp directly

Set a repetition penalty and stop strings, otherwise the model will keep writing its own turns (see the note below).

llama-server -hf respinosamena/Helios-Nova-306M-Instruct-GGUF:Q8_0 \
  -ngl 99 -c 2048 --port 8080 --temp 0.7 --top-k 40 --repeat-penalty 1.2

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role":"user","content":"What causes the seasons on Earth?"}],
  "temperature": 0.7, "top_k": 40, "repeat_penalty": 1.2, "max_tokens": 256,
  "stop": ["### User:", "### System:", "### Assistant:"]
}'

Note: this 306M model does not always emit its EOS token, so on its own it can continue by writing the next ### User: turn. Always set a repetition penalty (~1.2) and stop strings on the turn markers ["### User:", "### System:", "### Assistant:"]. The helios_chat.py client does this automatically.

Chat template

Plaintext markers (no special chat tokens were added during SFT). <|bos|> is prepended automatically and the template is baked into the GGUF, so llama-server's /v1/chat/completions formats prompts for you.

### System:
You are a helpful assistant.
### User:
What is the capital of France?
### Assistant:
The capital of France is Paris.

<p align="center"><img src="assets/architecture.png" alt="Architecture diagram" width="80%"/></p>

Underlying model performance

These quantizations inherit the capabilities of the base family. All models below were re-run through one identical lm-evaluation-harness configuration (0-shot).

<p align="center"><img src="assets/token_efficiency.png" alt="Capability versus pre-training token budget" width="70%"/></p>

96% of SmolLM2-360M on commonsense reasoning at ~80× less pre-training data. The model is data-efficient, not knowledge-rich, and trails on broad recall (TriviaQA) and exam-style knowledge (MMLU).

<p align="center"><img src="assets/base_benchmarks.png" alt="Full benchmark sweep" width="92%"/></p>

How these were made

Converted with llama.cpp (build b9442). The custom Helios Nova checkpoint was re-keyed to standard HuggingFace tensor names and run through convert_hf_to_gguf.py against a llama.cpp architecture that implements the same components (GQA with per-head QK-Norm before RoPE, SwiGLU, tied embeddings). No weights were permuted or modified. The exported model reproduces the original PyTorch logits with zero difference in f32, tokenizes identically, and greedy-decodes token-for-token; Q8_0 adds +0.07% perplexity and Q4_K_M +2.3%.

Limitations

Inherited from the base model: English only, 2,048-token context, no safety alignment, and a 306M capacity ceiling — good at short factual Q&A and simple instructions, weak at multi-step reasoning, code, and long multi-turn chats. It can confidently hallucinate. See the Instruct model card for details.

The Helios Nova family

Citation

@misc{espinosamena2026heliosnovagguf,
  title  = {Helios Nova 306M-Instruct (GGUF): quantized builds for llama.cpp},
  author = {Espinosa Mena, Rafael},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/respinosamena/Helios-Nova-306M-Instruct-GGUF}}
}

Contact

Rafael Espinosa Mena — [email protected]

License

Released under the Apache-2.0 license, inheriting from the base model. Copyright 2026 Rafael Espinosa Mena.