Nemotron-TheHobbit-30B-A3B

Nemotron-TheHobbit-30B-A3B

The Two Towers, shrunk to fit in one. 🧙

NVIDIA's Nemotron-Labs-TwoTower-30B-A3B-Base is a block-diffusion LM whose two-tower design officially needs 2× 80 GB (H100/A100) — ~59 GB per GPU in BF16 — to run its diffusion mode. This is that model put on a Config-I diet: a 4-bit TQ4_1S quantization that squeezes both towers into ~38 GB, so the full AR and mask-diffusion paths run on a single GPU. A little Hobbit that walks into Mordor where the Two Towers needed a fellowship.

Verified end-to-end on one NVIDIA DGX Spark (GB10, 128 GB unified).

⚠️ Research artifact — correct but slow. Runs in eager PyTorch with kernel-free torch fallbacks (no fused mamba_ssm/causal_conv1d), doing two datacenter-GPUs' work serially on one card: ~2–4 tok/s on a GB10. Shared so the community can test it on real hardware and speed it up — see Help wanted.

What "Config-I" means

Config-I is TurboQuant's TQ4_1S weight format: a Walsh–Hadamard (RHT) rotation applied per 32-element block, then 4-bit 16-level Lloyd-Max quantization with dual half-block scales (~5 bits/weight). The rotation is the trick — it Gaussianizes every weight distribution, so one fixed 4-bit codebook is near-optimal for every tensor type at once. That's what lets it survive the Mamba-2 state-space recurrence, where naïve int8 collapses into garbage.

Measured per-weight reconstruction cosine ≈ 0.997, uniform across Mamba-2, attention, and the 128-expert MoE tensors. SSM scalar params (A_log, dt, conv1d, D), norms, router gates, the PixArt-style adaLN time-conditioning tables, and embeddings are kept at BF16. Both AR and diffusion produce coherent, mutually-consistent text.

The model underneath

A two-tower block-diffusion LM: a context tower + a time-conditioned denoiser tower, each a Nemotron-H hybrid (Mamba-2 SSM + attention + 128-expert MoE, top-6, ~3 B active). It generates a block of tokens in parallel via iterative masked denoising instead of one token at a time.

Running it

pip install torch transformers accelerate safetensors einops numpy
python run_configi.py     # loads both towers on cuda:0, runs an AR + diffusion demo

run_configi.py is self-contained: it injects torch stand-ins for the mamba_ssm / causal_conv1d kernels the denoiser imports (so you do not need mamba-ssm, which won't build on many GPUs), swaps the quantized Linears for an on-the-fly Config-I dequant layer, and applies a sync-free MoE. See the script for the load() / generate_ar / generate_mask_diffusion API.

Performance & why it's slow

NVIDIA's throughput comes from 2× 80 GB with fused mamba_ssm/causal_conv1d Triton kernels. Here, on one GPU, the model is bottlenecked by the absence of fused kernels and by per-forward 4-bit dequantization. Speedups already applied — torch.compile'd dequant, a sync-free MoE (the stock loop did ~18 k CPU↔GPU syncs/token), and bf16 tensor-core dequant + autocast (the stock path fell to fp32 SIMT matmuls) — together ~9× (0.24 → ~2 tok/s diffusion). Datacenter throughput needs fused kernels and/or the 2-GPU reference setup.

Engine support status (July 2026 — 🚧 in progress)

Fast native inference for the two-tower diffusion path does not exist in llama.cpp or vLLM yet — the base model is only days old. As of this writing there is no TwoTower-specific support in either engine (searched both repos; zero "two-tower" PRs/issues). Current state:

• vLLM — no two-tower diffusion support. It does support the NemotronH AR backbone (NemotronHForCausalLM, PRs #47088, #47440), so vllm serve runs the autoregressive path, not the diffusion two-tower. Generic block-diffusion plumbing is landing via DiffusionGemma (#45982, #47462, #47464) — the infrastructure a TwoTower port would build on.
• llama.cpp — no two-tower / cross-attention-denoiser path. Block-diffusion is being added for DiffusionGemma (#24427) and LLaDA 2.0 (#17454). Any "GGUF quants for llama.cpp/Ollama/LM Studio" you see for Nemotron refer to the AR backbone (Nemotron-3-Nano), not this diffusion model.

vLLM and llama.cpp integration to support this is still in progress — will need to revisit. Until a fast engine path lands, NVIDIA's 2× 80 GB reference (fused mamba_ssm/causal_conv1d kernels) is the throughput path, and this repo's run_configi.py is the single-GPU (eager, ~2–4 tok/s) route. Whoever ports TwoTower to a fast engine will most likely build on the DiffusionGemma / LLaDA block-diffusion work landing now.

Help wanted

• Test on 2× 80 GB (or a single H200/B200) — report AR-vs-diffusion quality + throughput.
• fp8 weight cache — dequant once to fp8-resident (~63 GB, still 1 GPU) to kill per-forward dequant (est. ~10–15 tok/s).
• Fused kernels / CUDA graphs for the two-tower diffusion path.
• Retention benchmarks (MMLU / GSM8K / HumanEval, AR vs diffusion) vs the BF16 reference.

Reproduction

quantize_twotower_gpu.py (included) quantizes the BF16 checkpoint to Config-I on-GPU in ~7 min.

Attribution

• Base model: © NVIDIA, Nemotron-Labs-TwoTower-30B-A3B-Base, under the NVIDIA Open Model License. This is an independent community quantization, not affiliated with or endorsed by NVIDIA.
• Quantization: TurboQuant Config-I / tq4_1s (RHT-rotated 4-bit).
• Torch fallback kernels, sync-free MoE, and single-GPU loader added for this release.