gemma-4-12B-it-qat-NVFP4

gemma-4-12B-it-qat-NVFP4

An NVFP4-quantized version of google/gemma-4-12B-it-qat-q4_0-unquantized.

This model was independently quantized to NVFP4 W4A4 using LLM Compressor and saved in the compressed-tensors format.

Model Details

• Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
• Architecture: Gemma4UnifiedForConditionalGeneration
• Quantization: NVFP4 W4A4
• Serialization format: nvfp4-pack-quantized
• Compression backend: compressed-tensors
• Calibration samples: 1,024
• Maximum calibration sequence length: 8,192 tokens

Calibration Data

The following datasets were used for calibration:

• CohereLabs/aya_dataset
• interstellarninja/hermes_reasoning_tool_use

The calibration dataset consists of:

• 512 general multilingual instruction samples from Aya
• 512 reasoning and tool-use conversation samples from Hermes

Quantization Configuration

default_stage:
  default_modifiers:
    GPTQModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: false
            actorder: static
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: imatrix_mse
            observer_kwargs: {strict: true, expand: 1.8, maxshrink: 1 - 0.8 / 1.8, grid: 200.0, norm: 2.4, patience: 1000}
          input_activations:
            num_bits: 4
            type: float
            symmetric: true
            group_size: 16
            strategy: tensor_group
            block_structure: null
            dynamic: local
            actorder: null
            scale_dtype: torch.float8_e4m3fn
            zp_dtype: null
            observer: static_minmax
            observer_kwargs: {}
          output_activations: null
          format: null
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*vision.*', 're:.*audio.*', 're:.*router.*']
      kv_cache_scheme:
        num_bits: 8
        type: float
        symmetric: true
        group_size: null
        strategy: tensor
        block_structure: null
        dynamic: false
        actorder: null
        scale_dtype: null
        zp_dtype: null
        observer: static_minmax
        observer_kwargs: {}
      bypass_divisibility_checks: false
      requires_calibration_data: true
      block_size: 128
      dampening_frac: 0.01
      actorder: static
      offload_hessians: false

Reproducibility

The quantization code and calibration preprocessing are available at:

gemma4-nvfp4-quantization

The quantization run used:

transformers:        5.14.1
llmcompressor:       0.13.1.dev34+g040aa8cb3
compressed-tensors:  0.18.1a20260818