Downloads · 30 days
13
19% of all-time downloads
brazley/Brazley-Nemotron-3.5-Lightning-Omni-Drafter
Brazley-Nemotron-3.5-Lightning-Omni-Drafter is a machine learning model from brazley. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
Speculative drafter for Brazley-Nemotron-3.5-Lightning-Omni. Unofficial model created by Nik Brazley — not an NVIDIA release.
Downloads · 30 days
13
19% of all-time downloads
All-time downloads
68
Public
Parameters
764M
1.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB · 100%
How the weights are stored.
BF16560M · 73%
From the Hugging Face model README
Speculative drafter for Brazley-Nemotron-3.5-Lightning-Omni. Unofficial model created by Nik Brazley — not an NVIDIA release.
Extracted from NVIDIA's published Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark checkpoint. This repository contains the drafter and the plugin shim that attaches it; it is not a standalone model and will not load as one.
The shim is required. Without it the parent model still starts, serves, and reports healthy — it simply runs slower, with no error anywhere. If you are benchmarking the parent model and cannot reproduce its published throughput, check that the shim is loaded first.
Not an NVIDIA release
This is an independent derivative. It is not an official NVIDIA product, it is not endorsed by, affiliated with, sponsored by, or supported by NVIDIA Corporation, and NVIDIA has not reviewed, validated, or approved it.
"NVIDIA", "Nemotron", and "DGX Spark" are trademarks of NVIDIA Corporation, used here solely to identify the origin of the source model. No trademark rights are claimed and no affiliation is implied. For NVIDIA's official drafter, see nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark.
| Type | Speculative draft model |
| For | Brazley-Nemotron-3.5-Lightning-Omni |
| Size | 1.3 GB |
| Precision | W4A16_NVFP4. Note that config.json declares "dtype": "bfloat16" at top level; the operative algorithm is in quantization_config |
| Resolves in vLLM as | Qwen3DSparkModel — natively supported in vLLM 0.27.1 |
| Serving depth | num_speculative_tokens: 3 |
| Also contains | nano_omni_eagle3.py + nano_omni_eagle3-0.1.dist-info/ — the required shim |
| Release Date | 08/20/2026 |
A speculative draft model — a small model that proposes several tokens ahead, which the full model verifies in a single step. Accepted proposals are effectively free; rejected ones cost the verification.
Its dimensions close exactly against the parent's transplanted text tower: hidden 2688, vocab 131072, 32 heads / 2 KV, head_dim 128 on both sides, and eagle_aux_hidden_state_layer_ids [2, 6, 20, 30, 42, 52] against the parent's 52 layers. It was trained by NVIDIA against stock Nemotron 3.5 Lightning and runs here unmodified against a merged model.
It serves through vLLM's DSpark path, which resolves it automatically from its architectures field.
Governing Terms: OpenMDW License Agreement v1.1. See License.
Accelerating decode for the parent model. This drafter's dimensions are specific to that target; it is not a general-purpose Nemotron drafter.
Runtime Engine(s): vLLM 0.27.1 Supported Hardware: NVIDIA Blackwell (DGX Spark / GB10)
vLLM's DSpark path requires the target model to satisfy the SupportsEagle3 protocol. NemotronHForCausalLM declares it. The merged parent, however, resolves to NemotronH_Nano_VL_V2 — the Omni multimodal wrapper — which does not, because stock Nano Omni has no speculative-capable text tower. The parent has one only by virtue of the Lightning transplant.
SupportsEagle3 is a @runtime_checkable Protocol and supports_eagle3() is a plain isinstance(), duck-typed on five members. Everything downstream already handles a multimodal wrapper: the protocol's set_aux_hidden_state_layers walks get_language_model(), so does dspark/utils.py::load_dspark_model, and NemotronH_Nano_VL_V2.forward returns the text tower's output verbatim, so an aux-hidden-state tuple passes straight through.
The shim therefore ships as a standard vllm.general_plugins entry point, bind-mounted into the stock image. No forked engine, no rebuilt image, nothing to maintain.
nano_omni_eagle3-0.1.dist-info/entry_points.txt
[vllm.general_plugins]
nano_omni_eagle3 = nano_omni_eagle3:register
Two read-only bind mounts alongside the parent model's launch. Both the .py and the .dist-info/ are required — the .dist-info is what makes vLLM discover the entry point.
docker run -d --name omni-lightning \
--gpus all --shm-size 32g --network host --ipc=host \
-v "${WEIGHTS}:/models/Brazley-Nemotron-3.5-Lightning-Omni:ro" \
-v "${DRAFTER}:/models/Brazley-Nemotron-3.5-Lightning-Omni-Drafter:ro" \
-v "${DRAFTER}/nano_omni_eagle3.py:/usr/local/lib/python3.12/dist-packages/nano_omni_eagle3.py:ro" \
-v "${DRAFTER}/nano_omni_eagle3-0.1.dist-info:/usr/local/lib/python3.12/dist-packages/nano_omni_eagle3-0.1.dist-info:ro" \
omni-vllm:v0.27.1-gcc12-audio \
--model /models/Brazley-Nemotron-3.5-Lightning-Omni \
--served-model-name omni-lightning-nvfp4 \
--tensor-parallel-size 1 --trust-remote-code \
--gpu-memory-utilization 0.65 --max-model-len 262144 \
--reasoning-parser nemotron_v3 \
--speculative-config '{"method":"dspark","model":"/models/Brazley-Nemotron-3.5-Lightning-Omni-Drafter","num_speculative_tokens":3}' \
--port 8200
If you edit the shim, clear the bytecode cache. A read-only bind mount lets a
.pyand its.pycdisagree indefinitely, and neither one tells you which was executed.
A draft head can build with zero loaded weights and skip silently, presenting as "speculation is on" at roughly 0% acceptance. Loading weights took 8.88 seconds reports presence, not correctness.
The shim compares every directly-comparable draft tensor against the same tensor read from the checkpoint file. On a good load:
[nano-omni-eagle3] DRAFT WEIGHTS VERIFIED: 118 checkpoint tensors -> 92 draft
parameters (702.4M elements); 39/39 directly-comparable tensors match the
checkpoint EXACTLY, 0 mismatches. Acceptance numbers from this run are trustworthy.
It raises rather than serving on a mismatch, and raises rather than certifying when nothing was comparable:
if not compared:
raise AssertionError("DRAFT LOAD UNVERIFIABLE: ... NOT ONE was directly
comparable. Refusing to certify the drafter loaded -- an unverifiable
gate must not report success.")
if mismatched:
raise AssertionError("DRAFT WEIGHTS DID NOT LOAD: ... A drafter with unloaded
weights accepts ~0% and looks identical to a working one from outside.
Refusing to serve.")
Tensor comparison rather than a parameter count, because 118 checkpoint tensors become 92 parameters when quantized scales fuse — a count mismatch is expected and proves nothing, while an exact match on an unquantized 131072×2688 tensor cannot happen by chance.
Behavioural corroboration: a correctly loaded drafter shows ~66% position-0 acceptance with a clean positional decay.
Measured against the parent model with speculation off, identical conditions (max_tokens 2048, temperature 1.0, top_p 0.95, enable_thinking false, first request discarded, every configuration launched fresh):
| Depth | c=1 | vs off | c=8 | vs off | Accepted / step |
|---|---|---|---|---|---|
| off | 80.76 tok/s | — | 245.93 tok/s | — | — |
| n=2 | 81.01 | +0.3% | 283.36 | +15.2% | 1.124 |
| n=3 — shipped | 91.46 | +13.2% | 276.50 | +12.4% | 1.337 |
| n=4 | 81.52 | +0.9% | 266.18 | +8.2% | 1.482 |
| n=7 | 81.70 | +1.2% | 237.39 | −3.5% | 1.655 |
n=3 is a peak rather than a default: n=2 and n=4 both fall off it at concurrency 1, and n=7 — the drafter's native block size — is worse at concurrency 8. Accepted-tokens-per-step keeps rising with depth while served throughput falls, which is why the shipped depth is chosen on tok/s.
Per-position acceptance against the merged parent at n=3: 66.1 / 40.5 / 24.7 (greedy, c=1), within 0.2 points of the same drafter's figures against stock Lightning.
These depth-sweep rows were measured under an earlier shim revision whose gate verified one tensor rather than all comparable tensors; the acceptance figures on the parent model's card were taken under the current gate.
All figures are text-only.
<a id="drafter-license"></a>
This drafter is redistributed from
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
under the OpenMDW License Agreement, version 1.1.
The full text is included as LICENSE.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This repository contains no Omni-derived material, so the NVIDIA Open Model Agreement governing part of the parent model does not apply here and is deliberately not shipped with it — including it would misstate what this repository contains.
If you redistribute this drafter or a derivative of it, you must retain a copy of the OpenMDW-1.1 agreement and all copyright notices and notices of origin. The license terminates your grant if you initiate patent or copyright litigation concerning these materials.
No warranty. OpenMDW-1.1 provides these materials "AS IS", without warranty of any kind. This derivative is provided on the same basis.
The two files are covered by different grants. Which one applies depends on which file you are using, not on the repository as a whole.
| File | License | Holder |
|---|---|---|
model.safetensors, config.json, hf_quant_config.json | OpenMDW-1.1 — LICENSE | NVIDIA CORPORATION & AFFILIATES |
nano_omni_eagle3.py | Apache License 2.0 — LICENSE.shim-Apache-2.0 | Nik Brazley |
The shim is original work by Nik Brazley and contains no NVIDIA code — it declares an
existing vLLM protocol against an existing vLLM class and verifies loaded tensors against
the checkpoint file. It is released under the Apache License, Version 2.0, which
matches the license of vLLM itself, the project whose protocol it implements. The full
unmodified license text ships as LICENSE.shim-Apache-2.0.
Copyright 2026 Nik Brazley. Licensed under the Apache License, Version 2.0.
The file carries an SPDX header so the grant travels with it. That matters here more than usual: the normal way to use this shim is to drop it into a vLLM plugins directory, which separates it from every license file in this repository.