nomos-gemma-4-31b-nvfp4

Gemma-4-31B-IT — NVFP4 weights for the Nomos kernel

Kernel: github.com/lightofbaldr/nomos-nvfp4 (MIT) — a pure-Mojo inference kernel for Gemma-4-31B: 4-bit weights and activations, lossless speculative decoding via DFlash, tensor-core batch verify on NVIDIA Blackwell.

The exact weight set that kernel loads, converted from google/gemma-4-31b-it into the kernel's native NVFP4 layout. Published so that anyone running the kernel gets the same setup we run locally, without having to reproduce the conversion.

This model is not fine-tuned. It is a format conversion of Gemma-4-31B-IT. Nothing has been trained, merged, or distilled into it.

What this is

Verify you have the right bytes:

layers_0_self_attn_q_proj_weight.nvfp4   sha256 starts b4908eb2
layers_30_mlp_down_proj_weight.nvfp4     sha256 starts 3fa309a8
embed_tokens_weight.nvfp4                sha256 starts 64dbb6e2

Performance — reported straight

Single-stream decode, one card. Two numbers, because only quoting the good one would be misleading:

Base decode alone is slower than llama.cpp's mature Q4_0 and we are not going to pretend otherwise. The win comes from the speculative-decode stack, which reaches ~50 tok/s against llama.cpp's realistic single-stream ceiling of ~30 (1.67×). That comparison is stack-vs-stack: vanilla llama.cpp has no valid standalone Gemma-4-31B drafter, because the available drafters are coupled next-n heads with no independent K/V and therefore cannot be passed as --model-draft.

These are our measurements on our hardware. Reproduce them before relying on them.

Using it

These weights are for the Nomos kernel, not transformers. They will not load with AutoModelForCausalLM — the layout is the kernel's own flat NVFP4 format.

git clone https://github.com/lightofbaldr/nomos-nvfp4 && cd nomos-nvfp4
pixi install
bash refresh_build.sh

# this repo — target weights at the root, drafter in drafter/
hf download Adam1010/nomos-gemma-4-31b-nvfp4 --local-dir ~/nomos_data/gemma-4-31b-nvfp4
export WEIGHTS=~/nomos_data/gemma-4-31b-nvfp4/
export DFLASH_DIR=~/nomos_data/gemma-4-31b-nvfp4/drafter/   # TRAILING SLASH REQUIRED

./nomos bench            # base decode
./nomos bench --spec     # with speculative decoding
./nomos smoke --prompt "hello"

Serving:

export NOMOS_SERVE_SPEC=1
pyhost/serve_nomos.sh up
curl -s localhost:$NOMOS_PORT/health

Tokenizer is not included. Use the tokenizer from google/gemma-4-31b-it; the kernel loads it via TOK_DIR. Gemma-4 frames turns with <|turn> / <turn|> (ids 105/106) and has no <start_of_turn> — much published Gemma guidance describes the older scheme and does not apply to this model. Always render prompts with the tokenizer's own apply_chat_template.

The drafter in drafter/ is DFlash by Z Lab (Chen, Liang, Liu), Apache-2.0 — not our work; we converted the format and built the NVFP4-native lossless verify path. See drafter/README.md for credit, citation and terms.

Thinking channel: Gemma-4 relies on its thought channel. Suppressing it, or truncating it with a small max_tokens, produces output that looks like a numerics bug and is not one.

Contents

Licence

The target weights are governed by the Gemma Terms of Use. Use is subject to Google's Gemma Prohibited Use Policy. By downloading you agree to those terms. This is a derivative of Gemma-4-31B-IT; all rights in the underlying model remain Google's. Redistribution must carry these terms forward.

Provenance

Converted from the Google QAT release. The source weight set was independently verified across two machines before conversion, and the spot hashes above are the identity check we use internally — we had a silent weight-corruption incident once and now treat byte-level verification as mandatory rather than optional.