Gemma-4-E2B-it-TextOnly-GPTQ-a16w4

Gemma-4-E2B-it-TextOnly-GPTQ-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model architecture: Gemma 4 E2B (text-only)
• Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
• Pre-quantization: The source checkpoint was pre-quantized with GPTQ before compilation.
• Maximum context length: 4096 tokens
• Source model: google/gemma-4-E2B-it

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

hf download simaai/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 --local-dir /media/nvme/llima/models/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4

The model is installed under:

/media/nvme/llima/models/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 --local-dir Gemma-4-E2B-it-TextOnly-GPTQ-a16w4
scp -r Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Gemma-4-E2B-it-TextOnly-GPTQ-a16w4

For application integration, see GenAI with LLiMa.

Recompilation

To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.

The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• Quantization can cause minor differences from the full-precision source model.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• Source model