Qwen3-4B-Autoround-a16w4

Qwen3-4B-Autoround-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Qwen3-4B-Autoround-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model architecture: Qwen3 4B
• Quantization: A16W4 (16-bit activations, 4-bit weights)
• Pre-quantization: The source checkpoint was pre-quantized with AutoRound before compilation.
• Maximum context length: 8192
• Source model: Qwen/Qwen3-4B

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, five samples per input length, and up to 128 generated tokens. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

llima pull Qwen3-4B-Autoround-a16w4

The model is installed under:

/media/nvme/llima/models/Qwen3-4B-Autoround-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Qwen3-4B-Autoround-a16w4 --local-dir Qwen3-4B-Autoround-a16w4
scp -r Qwen3-4B-Autoround-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Qwen3-4B-Autoround-a16w4

For application integration, see GenAI with LLiMa.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• Quantization can cause minor differences from the full-precision source model.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• Source model