gemma-4-e2b-monarch-35mlp-int8

Gemma 4 E2B Distilled INT8

Experimental weight-only INT8 version of hexoy/gemma-4-e2b-distilled at immutable source revision f897353fca328b1cc5fd2e12d645773ca637f5f0.

All 420 supported, untied standard nn.Linear weights across language, vision, audio, and multimodal components use TorchAO 0.17.0 symmetric per-output-channel INT8 weight-only quantization. All 210 trained Monarch factor tensors, the tied embedding/LM-head weight, normalization parameters, biases, unsupported non-linear parameters, and runtime activations remain BF16.

Architecture And Weight Storage

The BF16 loaded footprints are parameter storage at two bytes per parameter. The quantized footprint was audited from physical TorchAO tensors. Serialized values count safetensor files only. None of these values includes activations, logits, temporary buffers, CUDA workspaces, or allocator overhead.

The INT8 model contains 779,419,648 quantized linear weights. The BF16 remainder contains 2,902,849,056 parameters (5.41 GiB), including 135,106,560 Monarch factor parameters. Quantized physical storage is measured from the TorchAO qdata, scale, zero-point, BF16 parameter, and buffer tensors; it is not the logical BF16 shape reported by generic PyTorch tensor-subclass accounting.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "hexoy/gemma-4-e2b-monarch-35mlp-int8"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

This export requires torchao==0.17.0 and contains custom modeling code, so review it before enabling trust_remote_code=True. INT8 applies to weights only; activations and logits remain BF16.

Related models: BF16 Distilled Gemma 4 and experimental LoRA r8.

TinyHellaSwag Benchmark

All rows used the same official tinyHellaswag protocol: 100 anchor examples, 10-shot prompts, character-length-normalized continuation likelihood, no chat template, seed 1234, BF16 activations, fixed batch size 32, and an NVIDIA RTX PRO 6000 Blackwell GPU.

Relative to BF16 Distilled Gemma 4, INT8 changed GP-IRT by -1.78 percentage points and raw accuracy by -1 point. The paired bootstrap 95% interval was [-3, 0] points, with two disagreements and McNemar p=1.0. Relative to dense, the INT8 deltas were -8.66 and -8 points ([-18, +2], McNemar p=0.1849).

Pinned model revisions:

• Original: 9dbdf8a839e4e9e0eb56ed80cc8886661d3817cf
• BF16 35-layer: f897353fca328b1cc5fd2e12d645773ca637f5f0
• INT8 weights: db56825e2e0de59115049d7109632b2f1ce80905

Complete results, paired comparisons, logs, environment metadata, inventory, and checksums are retained in a private artifact repository.

Prompted MNLI Benchmark

Prompted MNLI used all 9,815 examples from the pinned GLUE MNLI validation_matched split, mean continuation-token likelihood, a 128-token context, and no chat template.

The INT8 model was not evaluated on prompted MNLI. The 35-layer BF16 result exhibits severe class collapse: it predicted entailment for 9,701 of 9,815 examples, so macro F1 is more informative than its raw accuracy.

Limitations

• This is an experimental compression and quantization artifact, not an official Google model.
• TinyHellaSwag has only 100 anchor items; small accuracy changes have wide uncertainty.
• Weight-only INT8 reduces persistent weights, while activation-heavy peak VRAM falls by less.
• Evaluate the model on the intended task before deployment.

Attribution

Derived from google/gemma-4-E2B-it through the BF16 Monarch model. See NOTICE for the modification summary.