LFM2.5-VL-1.6B-GPTQ-a16w4

LFM2.5-VL-1.6B-GPTQ-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the LFM2.5-VL-1.6B-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model Architecture: LFM2.5-VL (1.6B parameters)
• Quantization: BF16 activations; decoder Linear weights use GPTQ symmetric INT4, group size 256. LM head: RTN symmetric INT8, per output channel. Vision Linear weights use RTN symmetric INT8 per output channel. The two multimodal-projector Linear layers remain BF16.
• Calibration: 512 complete multi-turn UltraChat conversations, up to 4,096 tokens; RTN stages are data-free.
• Maximum context length: 4096 tokens
• Input Resolution: 448x448 (Fixed)
• Source Model: LiquidAI/LFM2.5-VL-1.6B
• Language group size: 128 tokens
• Embeddings / KV cache: INT8 quantization enabled
• Filter sharing: automatic compiler default

Recompilation

The build uses Model Compiler 3.0.0. Compiler revisions and the source-weight SHA-256 are recorded in compilation.json.

To compile a matching prequantized checkpoint, activate the Model Compiler environment and run:

MODEL_DIR=/path/to/LFM2.5-VL-1.6B-GPTQ-Safetensors bash compile.sh

compile.sh uses 4K context, 16 workers and 448x448 image input. It keeps the default quantized embeddings, quantized KV cache and filter sharing. Set OUTPUT_DIR to change the output directory; extra arguments are forwarded to llima-compile.

To recreate the quantization policy from the original BF16 model, use a separate CUDA environment matching prequantization/versions.txt:

hf download LiquidAI/LFM2.5-VL-1.6B --revision 919fde3d022e3f90a4716006f993938ee8c2eb97 --local-dir original-bf16
python prequantization/quantize.py \
    --model-path original-bf16 \
    --output-dir LFM2.5-VL-1.6B-GPTQ-Safetensors

The effective recipe and calibration sample manifest are included under prequantization/. This reproduction uses the public llmcompressor API.

Compilation and local unpacking completed successfully. Text-only inference performance was measured on Modalix as reported below. Image-conditioned quality has not yet been validated for this compiled refresh.

8k context

To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.

The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.

Performance

Text-only generation

The following measurements exercise the language-model component without an image. Vision encoding and projection latency are not included, and input tokens refer only to text tokens.

Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Image + text generation

The following measurements use one 448 × 448 image at the model's compiled input resolution and a text prompt.

Measured end-to-end on Modalix using five fresh requests. Values are arithmetic means. TTFT includes image decoding and preprocessing, vision encoding and projection, language-model prefill, and the first generated token. Generation rate is measured after the first token.

Prerequisites

To run this model, you need:

1. SiMa.ai Modalix Device
2. SiMa.ai CLI: Installed on your Modalix device.
3. SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
4. Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device.

The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model assets from this repository directly to your device.

# Download the model to a local directory
hf download simaai/LFM2.5-VL-1.6B-GPTQ-a16w4 --local-dir /media/nvme/llima/models/LFM2.5-VL-1.6B-GPTQ-a16w4

Alternatively, you can download the compiled model to a Host and copy it to the Modalix device:

hf download simaai/LFM2.5-VL-1.6B-GPTQ-a16w4 --local-dir LFM2.5-VL-1.6B-GPTQ-a16w4
scp -r LFM2.5-VL-1.6B-GPTQ-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device.

Expected Directory Structure:

/media/nvme/llima/
└── models/
    └── LFM2.5-VL-1.6B-GPTQ-a16w4/   # The compiled model

Usage

Validate with LLiMa CLI

Run the model directly on Modalix:

llima run LFM2.5-VL-1.6B-GPTQ-a16w4

For an image-grounded CLI check, enter:

add image /path/to/image.jpg
Describe the image.

For all runtime options, run:

llima run -h

GenAI Demo Application

The GenAI demo application is separate from LLiMa installation. Use the GenAI Multimodal Assistant page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.

API Usage

To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in Serve GenAI Models.

For direct VLM calls without setting up a server, see Run a VLM.

Limitations

• Quantization: The small GPU prompt suite completed without detected looping, but it does not establish broad reasoning accuracy; an incorrect Pythagoras proof remained. Validate image quality and multi-turn behavior on Modalix.
• Fixed Resolution: While the standard LFM2.5-VL architecture supports dynamic input resolutions, this version has been specifically optimized and fixed to 448x448 resolution at compile time to achieve maximum throughput and efficiency on the SiMa.ai MLA.

Troubleshooting

• sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
• llima not found: Install or update the Neat Library. See Getting Started.
• Model can't be run: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/LFM2.5-VL-1.6B-GPTQ-a16w4/LFM2.5-VL-1.6B-GPTQ-a16w4).
• Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run a VLM
• GenAI Multimodal Assistant