Lightning-Herald-30B-A3B

Lightning-Herald-30B-A3B

<p align="center"> <img src="Lightning-Herald-30B-A3B.jpg" alt="Lightning-Herald" width="420"> </p>

A Hermes-Agent tune of NVIDIA's Nemotron 3.5 Lightning. 30B total, 3B active, so it runs an always-on local agent on a single RTX 3090 and stays quick.

I like Lightning a lot. It's a hybrid (Mamba-2, MoE and a bit of attention), NVIDIA put it out under OpenMDW-1.1, and it's fast enough to sit under an agent loop all day without you noticing. What I wanted was the version of it that drives the Hermes Agent harness best. So I fine-tuned it on a filtered set of real Hermes trajectories, but with one twist that I think matters more than anything else in the recipe: every trace was re-rendered into Nemotron's own chat template and tool-call syntax before training. The model learns exactly the format llama.cpp's parser wants to see. No borrowed Kimi or GLM formatting, no format confusion.

Built and benchmarked over a weekend on two 3090s.

GGUF builds are in EryriLabs/Lightning-Herald-30B-A3B-GGUF.

The numbers

This is stock Lightning against this tune, and nothing else. Same quant (IQ4_XS, and I built both files with the same convert-and-quantize pipeline so neither side gets a fancier quant), same llama.cpp build, same sampling (temp 1.0, top-p 0.95, which is what NVIDIA's card recommends), same harness, same task set. Only the weights differ. I make no claims against any other model.

Per-run scores, because agent evals are noisy and you should see the spread:

• Fox. Stock: 84, 88, 92, 84, 84. Herald: 92, 92, 88, 88, 96.
• Terminal. Stock: 100, 93, 93, 87, 93. Herald: 93, 93, 93, 93, 93.

My honest read: the fox gain is real. Every one of Herald's five runs lands at or above stock's median, and the gain sits exactly where the training data aims, which is multi-step tool use. The terminal suite is close to ceiling for both models, so I treat it as a no-regression check rather than a headline. No cherry-picking. The eval scripts and the raw per-task results are in the eval/ folder of this repo. Run them yourself.

What actually went wrong, before and after

I went through the transcripts and counted failure modes. Totals over 5 runs of each suite:

Read it straight. Herald makes fewer bad tool calls and declares victory early less often. That's the training doing its job. It also runs into the iteration cap a bit more on the terminal tasks: it would rather keep working a problem than give up, which is the right instinct, but occasionally it doesn't wrap up in time. That's on my list for v2.

Running it

llama-server -m Lightning-Herald-30B-A3B-IQ4_XS.gguf --jinja -ngl 999 -c 32768 --temp 1.0 --top-p 0.95

IQ4_XS is the exact file the table was measured with and it fits on one 24GB card. Q4_K_M, Q5_K_M, Q6_K and Q8_0 are in the GGUF repo if you have more room.

--jinja is the important flag. It makes llama.cpp render Nemotron's chat template with your tools and parse the model's native tool-call syntax into proper OpenAI-style tool_calls. Then point Hermes Agent (or anything OpenAI-compatible) at http://127.0.0.1:8080/v1. There's a ready-made Hermes config in hermes-preset/.

Only 6 of the 52 layers are attention, so the KV cache is tiny. Long agent sessions are cheap and -c 65536 fits comfortably at Q4 on 24GB.

Training details

Data. DJLougen/hermes-agent-traces-filtered (Apache 2.0). That's 3,679 reasoning-heavy Hermes trajectories, a strict subset of lambda/hermes-agent-reasoning-traces. The traces were recorded from Kimi and GLM driving the harness, so I parsed each one into a neutral schema and re-rendered it with Nemotron's tokenizer chat template: tools passed structurally, reasoning into <think>, tool calls into the native <tool_call><function=...> XML. Then I round-tripped every rendered example back through the same parsing rules llama.cpp uses. 100% of tool calls had to re-parse before I'd spend any GPU time; the handful that failed were dropped, not patched. Trajectories over 16k tokens keep their longest turn-prefix that ends on an assistant turn. Final split: 3,193 train, 168 held out. I trained on a 1,946-row subset (about 22M tokens) to keep the run inside a few hours.

Loss masking. Assistant tokens only. Tool outputs and user turns are masked out, so the model learns to produce actions rather than imitate tool output.

Trainer. Axolotl 0.18, QLoRA. load_in_4bit with quantize_moe_experts so the routed experts sit frozen in NF4. LoRA r=32, alpha=64, no dropout, on the attention q/k/v/o projections and the always-active shared-expert up/down. I left the router, the Mamba-2 mixers, the routed experts and the MTP tower alone: they're products of NVIDIA's training, cheap to break and expensive to fix. Sequence length 16384, context-parallel across the two 3090s, lr 2e-4 cosine, one epoch, effective batch 16, 95 optimizer steps.

Merge-time alpha sweep instead of retraining. I merged W = W_base + alpha * delta for alpha in {0.5, 0.75, 1.0}, quantized each the same way and benchmarked all three identically. Alpha 0.75 won and gave the most stable scores of the sweep. That's what's released here. The LoRA adapter is in adapter/ if you want to redo the sweep or try other values.

Data sources and licences

No non-commercial data went in. I looked at Simia-Tau and left it out because of its NC licence.

Responsible use

This is an agent driver. Given a harness, it will run shell commands and call tools. Run it in a sandbox, look at what it plans to execute, and keep the usual guardrails on. It inherits Nemotron's general behaviour and limits; the tune targets agentic format and process, not knowledge.

Reproduce the eval

Everything you need is in eval/: the fox 25-task suite, HeraldBench-Terminal, the lane runner (serve a GGUF, run both suites) and the aggregator. Pin llama.cpp to b10413 or newer.

Roadmap

• A LoRA soup with a Simia-Tau-formatted adapter, which would have to ship as an NC release
• GRPO on Hermes trajectories via Atropos
• An ablation that includes the router and Mamba layers

---

Dwain Barnes, EryriLabs