Lightning-Herald-30B-A3B-GGUF

Lightning-Herald-30B-A3B GGUF

<p align="center"> <img src="Lightning-Herald-30B-A3B.jpg" alt="Lightning-Herald" width="360"> </p>

GGUF builds of EryriLabs/Lightning-Herald-30B-A3B, my Hermes-Agent tune of NVIDIA Nemotron 3.5 Lightning (30B total, 3B active). The full model card, benchmarks and training details live on the main repo. This one is just the files.

Files

All made with llama.cpp b10413 (convert_hf_to_gguf.py then llama-quantize) from the released BF16 weights. You need a llama.cpp build with Nemotron-H support, so b10413 or newer.

Run

llama-server -m Lightning-Herald-30B-A3B-IQ4_XS.gguf --jinja -ngl 999 -c 32768 --temp 1.0 --top-p 0.95

--jinja matters: it makes llama.cpp render the chat template with your tools and parse tool calls into structured tool_calls. The sampling above is NVIDIA's recommendation for the base model and what the benchmark used.

Point Hermes Agent at http://127.0.0.1:8080/v1. A config preset is on the main repo.

Licence

OpenMDW-1.1, inherited from the base. Training data was Apache 2.0.

---

Dwain Barnes, EryriLabs