Nemotron-Labs-Diffusion-3B-JQ4

Nemotron-Labs-Diffusion-3B

---

Deliverance Quantization

This model directory was generated by Deliverance Q.O.D. (Quantize On Demand) from a local source model.

Deliverance rewrote selected matrix weights into its Q4 quantized tensor format to reduce model size and improve local inference throughput while keeping the model card and tokenizer metadata intact.

What Changed

• Original local size: 7.2 GB
• Quantized local size: 3.3 GB
• Target dtype: Q4
• Quantized tensors: 182
• Provenance manifest: deliverance-quantization.json

The manifest records the source/output paths, tensor dtype changes, generated .qb sidecars, and shape-normalization transforms used during export.

Try Deliverance

Deliverance is a local Java inference engine with safetensors loading, Q4 quantization, and quantize-on-demand model generation. Learn more at github.com/edwardcapriolo/deliverance.

<div align="center" style="line-height: 1;"> <a href="https://d1qx31qr3h6wln.cloudfront.net/publications/Nemotron_Diffusion_Tech_Report.pdf?VersionId=1tm4XZATEzGV7cs51XAf.xmWupU20vYW" target="_blank" style="margin: 2px;"> <img alt="Chat" src="https://img.shields.io/badge/📝Paper-Read Now!-536af5?color=76B900&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/collections/nvidia/nemotron-labs-diffusion" target="_blank" style="margin: 2px;"> <img alt="Nemotron-Labs-Diffusion Model Family" src="https://img.shields.io/badge/%F0%9F%A4%97-Nemotron--Labs--Diffusion_Model_Family-76B900" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://github.com/NVlabs/Nemotron-Labs-Diffusion" target="_blank" style="margin: 2px;"> <img alt="GitHub" src="https://img.shields.io/badge/GitHub-Github Repository-76B900?logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/" style="margin: 2px;"> <img alt="License" src="https://img.shields.io/badge/License-NVIDIA Open Model License-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/> </a> </div>

Model Overview

Nemotron-Labs-Diffusion is a tri-mode language model that supports both AR decoding and diffusion-based parallel decoding by simply switching the attention pattern of the same model during inference. The synergy between these two modes enables a third mode, called self-speculation: the same model performs diffusion-based parallel drafting and AR verification with shared KV cache, achieving high acceptance lengths and decoding efficiency. The seamless mode switching by simply changing attention patterns enables high efficiency at different concurrency levels in varying deployment scenarios with one single model.

<div align="center"> <img src="./assets/teaser.png" alt="An illustration of Tri-Mode LMs" width="500"> </div>

Highlights

• SOTA 3B, 8B, 14B dense LM family (base, instruct, and vision-language variants) supporting AR, diffusion, and self-speculation with the focus on decode efficiency.
• Generation moved from a memory-bound regime toward a compute-bound regime. Model weights are loaded once and reused to compute multiple tokens during generation.
• Self-speculation uses diffusion for drafting and AR for verification, providing a stronger alternative to MTP approaches:
  • 3x higher acceptance length and 2.2x speed-up vs. Qwen3-8B-Eagle3 in SGLang.
  • 5.9× tokens per forward over Qwen3-8B (no MTP) with the same accuracy.
• Real-device speed-up across platforms:
  • DGX Spark (8B, concurrency 1): 2.7x faster with 112 tok/sec vs. 41.8 tok/sec AR using w4a16.
  • GB200 (8B, concurrency 1): 3.3x faster with 850 tok/sec vs. 253 tok/sec AR and 360 tok/sec Eagle3. Custom CUDA kernels boost to 1015 tok/sec (4x).
• Diffusion speedup-of-light analysis shows that throughput can be further doubled (vs. current best) for a single user with better sampling - future research.

<div align="center"> <img src="./assets/result_acc.png" alt="Efficiency Results" width="800"> </div> <div align="center"> <img src="./assets/result_efficiency.png" alt="Acc Results" width="800"> </div>

License/Terms of Use

Use of this model is governed by the NVIDIA Nemotron Open Model License.

Environment

transformers>=5.0.0

Chat with Our Model

from transformers import AutoModel, AutoTokenizer
import torch

repo_name = "nvidia/Nemotron-Labs-Diffusion-3B"

tokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_name, trust_remote_code=True)
model = model.cuda().to(torch.bfloat16)

history = []

user_input = input("User: ").strip()
history.append({"role": "user", "content": user_input})

prompt = tokenizer.apply_chat_template(history, tokenize=False, add_generation_prompt=True)
prompt_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device='cuda')

## Chat in AR Mode
out_ids, nfe = model.ar_generate(inputs.input_ids, max_new_tokens=512)

## Chat in dLM Mode
out_ids, nfe = model.generate(prompt_ids, max_new_tokens=512, block_length=32, threshold=0.9, eos_token_id=tokenizer.eos_token_id)

## Chat in Linear Self-Speculation Mode
out_ids, nfe = model.linear_spec_generate(prompt_ids, max_new_tokens=512, block_length=32, eos_token_id=tokenizer.eos_token_id)

tokenized_out = tokenizer.batch_decode(out_ids[:, prompt_ids.shape[1]:], skip_special_tokens=True)[0]
print(f"Model: {tokenized_out}")
print(f"[Num Function Eval (NFE)={nfe}]")

Inference with Linear Self-Speculation + LoRA-enhanced Drafter

An optional LoRA adatper can be applied to the diffusion drafter in the linear self-speculation mode to further increase the acceptance length:

import torch
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel

repo = "nvidia/Nemotron-Labs-Diffusion-3B"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True)
model = model.cuda().to(torch.bfloat16)

# Attach the linear_spec LoRA adapter.
model = PeftModel.from_pretrained(model, repo, subfolder="linear_spec_lora").eval()
# Unwrap so we can call linear_spec_generate directly (it toggles LoRA internally).
base = model.model

history = [{"role": "user", "content": "Solve: What is 15% of 240?"}]
prompt = tokenizer.apply_chat_template(history, tokenize=False, add_generation_prompt=True)
prompt_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()

out_ids, nfe = base.linear_spec_generate(
    prompt_ids, max_new_tokens=512, block_length=32,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(out_ids[0, prompt_ids.shape[1]:], skip_special_tokens=True))
print(f"[NFE={nfe}]")

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the bias, explainability, safety & security, and privacy subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Citations

@techreport{fu2026nemotronlabsdiffusion,
  title       = {Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding},
  author      = {Yonggan Fu and Lexington Whalen and Abhinav Garg and Chengyue Wu and Maksim Khadkevich and Nicolai Oswald and Enze Xie and Daniel Egert and Sharath Turuvekere Sreenivas and Shizhe Diao and Chenhan Yu and Ye Yu and Weijia Chen and Sajad Norouzi and Jingyu Liu and Shiyi Lan and Ligeng Zhu and Jin Wang and Jindong Jiang and Morteza Mardani and Mehran Maghoumi and Song Han and Ante Jukic and Nima Tajbakhsh and Jan Kautz and Pavlo Molchanov},
  institution = {NVIDIA},
  year        = {2026},
  note        = {Technical report}
}