nemotron3-nano-kd-distilled

Nemotron 3 Nano 30B — KD Distilled

NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 fine-tuned with LoRA on 3,872 reasoning traces distilled from DeepSeek-V4-Flash (284B), with the adapter merged into the base weights.

Held-out pass@1 on coding problems: 23.0% → 63.4%, which is 92.5% of the teacher's own score at 9.5× fewer parameters.

Corpus construction, training, merging, and evaluation all ran on a single HP ZGX Fury (one NVIDIA GB300 Grace Blackwell Ultra). No cloud APIs at any stage.

Results

213 held-out problems (182 APPS, 8 HumanEval+), unseen during training. Greedy decoding.

What actually changed

The gain is about finishing, not about better algorithms.

The base model failed 138 of 213 problems by never producing code at all. It would analyse the problem correctly, describe a sound approach, and then run past the token budget mid-thought. Only 34% of its responses terminated on their own, at a mean of 9,736 tokens.

The teacher terminates every time, in 875 tokens. This model lands in between: 93% termination at 3,125 tokens. What transferred was the decision to stop analysing and commit to an answer, which matters most on hard problems, where the base model spirals longest.

On easy problems it matches the teacher exactly.

Usage

Loads and serves like the base model.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "curtburk/nemotron3-nano-kd-distilled"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True,
)

messages = [{"role": "user", "content": "Write a function returning the nth Fibonacci number."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
out = model.generate(ids, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

With vLLM:

vllm serve curtburk/nemotron3-nano-kd-distilled \
    --trust-remote-code --max-model-len 16384 --gpu-memory-utilization 0.85

Requires mamba-ssm and causal-conv1d, same as the base model. Generating with transformers directly may warn about NemotronHHybridDynamicCache; pass use_cache=False or use vLLM.

Training

Merged adapters cover attention, Mamba in_proj/out_proj, and MoE shared experts. Merging the routed expert adapters as well was tested and scored 8 points lower (55.4% vs 63.4%) — the model more often produced a complete analysis and then stopped without writing a solution. See the repo for details.

Note: Megatron-Bridge's save_hf_pretrained() does not merge subclass-style LoRA adapters and fails silently, producing an export byte-identical to the base model. The merge here was done in-place before export. See the repo for details and a working script.

Caveats

Single run. One seed, one checkpoint, one benchmark family. The direction held across every slice measured, but treat the exact magnitude as approximate.

Teacher score is a contaminated ceiling. The teacher generated traces for these same 213 problems during corpus construction, so the split is held out from the student but not from the teacher. Retention is approximate. The teacher also ran at a smaller token budget (10,240 vs 12,288) and on a different vLLM build.

Sampling mismatch. Evaluation used greedy decoding; the corpus was generated at temperature=1.0, top_p=1.0, which is what the teacher's model card recommends.

MBPP excluded. MBPP scored 1/23 for every model tested including the 284B teacher, which is a harness bug rather than a model property. It is left out of the table above rather than reported as a real number.

Format adherence did not transfer. The corpus uses a fixed markdown structure (Analysis / Approach / Implementation Plan / Solution / Verification). This model reproduces the full structure on only 2.3% of responses. It learned termination behaviour, not formatting.

Not evaluated outside coding. No measurement of whether general capability, chat quality, or long-context behaviour regressed. Assume they may have.

Licenses

Base model under the NVIDIA Open Model License. Training traces generated from DeepSeek-V4-Flash, which is MIT-licensed.

Links

• Training corpus: curtburk/nemotron3-nano-kd-corpus
• Pipeline and evaluation harness: https://github.com/curtburk/distillation-nemotron3