Phi-4-mini-instruct-Autoround-a16w4

Phi-4-mini-instruct-Autoround-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Phi-4-mini-instruct-Autoround-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model architecture: Phi-4 Mini
• Quantization: A16W4 (16-bit activations, 4-bit weights)
• Pre-quantization: The source checkpoint was pre-quantized with AutoRound before compilation.
• Maximum context length: 8192
• Source model: microsoft/Phi-4-mini-instruct

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, five samples per input length, and up to 128 generated tokens. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

llima pull Phi-4-mini-instruct-Autoround-a16w4

The model is installed under:

/media/nvme/llima/models/Phi-4-mini-instruct-Autoround-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Phi-4-mini-instruct-Autoround-a16w4 --local-dir Phi-4-mini-instruct-Autoround-a16w4
scp -r Phi-4-mini-instruct-Autoround-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Phi-4-mini-instruct-Autoround-a16w4

For application integration, see GenAI with LLiMa.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• Quantization can cause minor differences from the full-precision source model.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• Source model