translategemma-12b-it-NVFP4A16

henry1477/translategemma-12b-it-NVFP4A16

NVFP4A16 quantization of google/translategemma-12b-it (revision d1b225e1caa17f1ddc7e62065d8637d0923f34e2), produced with llm-compressor and intended for vLLM on Blackwell.

Weight-only NVFP4 (W4A16): 4-bit NVFP4 weights with group size 16 and FP8 block scales, 16-bit activations. This is the variant to use for structured output -- it preserves the model's ability to emit valid JSON, at roughly an order of magnitude more throughput than an int8 CPU-offloaded baseline.

What is different about this checkpoint

• Text-only. The SigLIP vision tower and multimodal projector are removed, not merely ignored: 12.19B -> 11.77B parameters, and the architecture is Gemma3ForCausalLM. Subtitle translation never uses the image path, and dropping it returns ~0.8 GiB of VRAM to the KV cache, which is the binding constraint on concurrency for this model on a 16 GB card.

• GPTQ with multilingual, in-format calibration. 512 calibration prompts drawn from Helsinki-NLP/opus-100 across 30 source languages (ar bg ca cs da de el es fi fr he hi hu id it ja ko ms nl no pl pt ro ru sv th tr uk vi zh), filtered to spoken-dialogue length (8-90 chars) and rendered as JSON cue arrays under this model's own chat template -- the exact shape a subtitle pipeline sends. GPTQ builds its Hessian from activations on those prompts, so calibrating on bare sentences would optimize for a distribution the model never sees in this role. Targets are split between en and es-MX.

  This is not cosmetic. Measured on a real 30-scene Spanish track, moving from data-free RTN to GPTQ with this calibration took structural pass from 77% to 100% and recovered 581/581 cues instead of 460, with chrF++ against the reference decode edging up (84.44 -> 84.64). English-only or data-free calibration leaves accuracy on the table for multilingual models; see Calibrating Beyond English.

• lm_head is excluded from quantization (Gemma3 ties it to the embedding table).

Measured results

Corpus: 120 real Spanish subtitle cues (6 scenes of 20), es->en, on an RTX 5070 Ti (sm_120, 16 GB). "Structural pass" is the fraction of scenes whose output is a valid JSON array carrying every requested cue id -- the contract a subtitle pipeline needs in order to map translations back onto timings. "chrF++" is measured against a bf16-adjacent int8 decode of the same cues.

The transformers rows are single-stream (batch 1); the vLLM rows are aggregate throughput across a batched run (6 scenes for the W4A4 rows, 30 for the A16 rows). They are therefore not a like-for-like latency comparison -- the honest summary is that the vLLM+NVFP4 path turns a stage that was bottlenecked on fp32 CPU-offloaded layers into one that saturates the GPU.

chrF++ is not reported for the W4A4 rows: with most scenes failing to parse there are too few recovered cues to score meaningfully. Where W4A4 output did parse, the translations themselves were good -- the failure is structural, not linguistic.

The W4A4 caveat matters. Both W4A4 variants produced fluent, accurate translations while dropping the requested JSON structure -- cue ids simply gone. A publicly available uncalibrated quant lost it on 6 of 6 scenes; the in-domain calibrated build in this collection recovers only part of the gap (2 of 6). Weight precision is identical across W4A4 and W4A16 and the throughput difference is under 5%, so quantizing activations to 4 bits buys almost nothing here and costs the output contract.

Calibration data is therefore worth something but is not a fix: it moved structural pass from 0% to 33% while W4A16 reaches 83% on the same corpus, same base revision, same remap. If your use depends on a specific output format, either constrain the decoder (see below) or use the A16 variant. A throughput-only benchmark will not catch this -- under free decoding W4A4 looks marginally faster right up until nothing parses.

Grammar-constrained decoding changes the structural picture

The structural-pass column above is measured with free decoding. vLLM can instead constrain generation to a JSON schema (structured_outputs=), which makes malformed output impossible rather than detected after the fact. Pinning every array position to its cue id -- so the decoder chooses only the translation text -- was measured on the same 30-scene Spanish track (581 cues), on both checkpoints, back-to-back in a single process:

Three things follow:

• Constraining costs about 6% throughput and is essentially output-neutral on the A16 checkpoint: 563 of 581 cues come back byte-identical to the free decode (chrF++ 98.9). It is a guarantee, not a distortion.
• It removes the W4A4 structural failure entirely -- 33% -> 100%, with all 581 cues recovered instead of 200. That failure is a decoding-time formatting problem, not lost capability.
• Constraining does not paper over a quality gap, and does not close one either: W4A4's agreement with the A16 decode is about the same whether free (83.9, over only the 200 cues it managed) or pinned (83.3, over all 581), and both stay far below A16's 98.9 agreement with itself.

The free-decode structural numbers in the table above are therefore a property of the checkpoint and the decoding setup, not of the weights alone. Absolute tok/s in this table is not comparable with the one above -- the two runs differ in compile-cache warmth -- so compare within this table only.

For this checkpoint constrained decoding is optional: free decoding already passes 30/30. Reach for it when you need a hard guarantee rather than a high success rate. It is cheap, and it does not change what the model says.

Usage (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(model="henry1477/translategemma-12b-it-NVFP4A16", max_model_len=4096, gpu_memory_utilization=0.85)

To constrain the output to a JSON schema (see the section above), pin each array position to the id you asked for so the decoder chooses only the text:

from vllm.sampling_params import StructuredOutputsParams

schema = {"type": "array", "minItems": len(ids), "maxItems": len(ids),
          "prefixItems": [{"type": "object", "additionalProperties": False,
                           "required": ["id", "text"],
                           "properties": {"id": {"const": i},
                                          "text": {"type": "string", "minLength": 1}}}
                         for i in ids],
          "items": False}
params = SamplingParams(max_tokens=1024, temperature=0.0,
                        structured_outputs=StructuredOutputsParams(json=schema))

On consumer Blackwell (sm_120: RTX 5070 Ti / 5080 / 5090, RTX PRO 6000) set these before importing vllm, or the FP4 GEMM raises an illegal instruction:

export FLASHINFER_CUDA_ARCH_LIST=12.0f
export FLASHINFER_FORCE_SM=120f
export FLASHINFER_DISABLE_VERSION_CHECK=1

Verify the engine log selects CutlassNvFp4LinearKernel. If it selects a Marlin NVFP4 path instead, output can be silently empty on sm_12x.

FlashInfer JIT-compiles kernels on first use, so the container needs ninja and a reachable nvcc; CUDA_HOME must point at vLLM's bundled CUDA (site-packages/nvidia/cu13) on a CUDA runtime base image.

Note that in plain transformers there is no FP4 kernel, so this checkpoint is slower than bf16 outside vLLM.

Recipe

default_stage:
  default_modifiers:
    GPTQModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: NVFP4A16
      bypass_divisibility_checks: false
      requires_calibration_data: true
      block_size: 128
      dampening_frac: 0.01
      actorder: static
      offload_hessians: false

License

Governed by the Gemma Terms of Use; you must accept them on the base model before use. This is a derivative of google/translategemma-12b-it @ d1b225e1caa17f1ddc7e62065d8637d0923f34e2.