dana-nemo-4b

dana-nemo-4b

nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 with a LoRA adapter statically merged into the FP8 checkpoint, specialised for Building Management System (BMS) / HVAC question answering. It is a reasoning model: the chat template pre-opens a <think> block, so the model emits only a closing </think>, which is why it needs the reasoning parser shipped in this repo rather than a stock one.

This repo contains the merged weights — no adapter needs to be applied at load time.

Read Required sampling settings before serving. The default repetition_penalty is wrong for this model in a way that is easy to miss: it does not crash, it silently emits JSON at you on prose questions.

Model details

The reference deployment serves --max-model-len 130944, not the full 262,144, to fit the KV cache on a 24 GB GPU. That is a deployment choice, not a model limit.

Provenance

Adapter: peft LoRA, r=32, alpha=64, dropout=0.0 (scaling = alpha/r = 2.0), targeting q,k,v,o,up,down_proj. Trained against the BF16 base, then merged into the FP8 checkpoint.

Required sampling settings

Serve with:

{"repetition_penalty": 1.05, "temperature": 0.0}

This is not a stylistic preference. vLLM's repetition_penalty is the only one of the three penalties that counts tokens in the prompt as well as in the generated text. The chat template injects a "think nudge" containing </think> nine times, so a penalty of 1.1 penalises the exact tag the prompt requires the model to emit.

Never send frequency_penalty or presence_penalty. A request carrying frequency_penalty=0.3 killed the engine with a CUDA device-side assert (EngineDeadError). Reject them at your gateway rather than relying on callers.

Serving with vLLM

Pinned and verified against vLLM 0.27.1 / torch 2.13.0+cu130. vLLM 0.27.1 registers NemotronHForCausalLM; older versions do not.

vllm serve /path/to/dana-nemo-4b \
  --served-model-name dana-nemo-4b \
  --trust-remote-code \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --max-model-len 130944 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 4096 \
  --reasoning-parser-plugin /path/to/dana-nemo-4b/dana_reasoning_parser.py \
  --reasoning-parser dana \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --override-generation-config '{"repetition_penalty": 1.05, "temperature": 0.0}' \
  --enable-prefix-caching \
  --enforce-eager \
  --host 127.0.0.1 --port 8000

Notes on the non-obvious flags:

• --enforce-eager — keep it. CUDA graphs measured 2.4× faster (107 vs 45 tok/s) and then killed the engine with cudaErrorLaunchFailure under concurrent load.
• --kv-cache-dtype fp8 uses the calibrated scales in hf_quant_config.json (kv_cache_quant_algo: FP8), so there is no uncalibrated-scale warning.
• --enable-prefix-caching measured an 84.8% hit rate in production.

Reasoning parser

dana_reasoning_parser.py registers as dana. It exists because the chat template pre-opens <think> in the prompt:

• nemotron_v3 assumes the model opens its own <think> — it cannot delimit a closing-only tag.
• nano_v3 handles a pre-opened block, but when thinking is disabled by the server default it files the whole reply as reasoning and leaves content null.

This parser keys off the output instead: no closing tag means no reasoning happened, so the text is the answer. It also keeps only the text after the last </think>, because the model sometimes closes the block more than once.

If you set DANA_THINK_END_STR to a bare "</think>", you trade the nudge for exact end-of-thinking detection used by thinking_token_budget.

Limitations

• Streaming + thinking-on can misplace text. extract_reasoning_streaming must classify each delta immediately; before any </think> arrives it cannot know whether one ever will, so if the model never closes the block that text stays in reasoning. No text is lost. Clients streaming with thinking on should read content or reasoning. Measured: thinking-off is 8/8 correct streaming and non-streaming — prefer it.
• Domain framing is needed. Unprompted, "low delta-T" is read as body temperature / hypothermia rather than as the HVAC fault. Add HVAC framing to the system prompt.
• Volunteered JSON. Even at repetition_penalty 1.05 the model offers JSON on roughly 10–30% of prose questions (non-deterministic). The reference deployment puts a proxy in front that appends a steering line to the system message and injects response_format={"type":"json_object"} only when the question actually asks for JSON. That JSON/prose contract is not a property of the weights.
• Training-data artifact. 160 of the 2149 distillation records carry a literal </s> in the target text, captured from a server running the 1.1 penalty. This is a known contaminant in the adapter's training set.
• English only; evaluated on building-management and HVAC questions.

License

Governed by the NVIDIA Nemotron Open Model License, inherited from nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8. Review the license terms before use. The LoRA adaptation and merge do not alter the base model's licensing.