LFM2.5-VL-3B-AWQ

LFM2.5-VL-3B-AWQ

AWQ (W4A16) quantization of LiquidAI/LFM2.5-VL-3B — a ~3B vision-language model from Liquid AI (lfm2_vl, native in transformers): a SigLIP2 NaFlex 400M vision encoder + the LFM2.5-2.6B hybrid conv+attention language backbone. Only the LFM2 language backbone is quantized here (self-attention + feed-forward projections); the SigLIP2 vision encoder, the short-convolution operators, the vision→LM projector, and all norms/embeddings are kept in BF16.

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Quantized by: sahilchachra Tooling: llm-compressor (AWQModifier + QuantizationModifier) -> compressed-tensors pack-quantized

This is a quantized derivative. Weights, behavior, and license follow the base model — see the original card for full details, benchmarks, and citation.

What is quantized

Quantized to 4-bit:

• LM backbone model.language_model.layers.*.self_attn.{q,k,v,out}_proj (8 full-attention layers)
• LM backbone model.language_model.layers.*.feed_forward.{w1,w2,w3} (all 30 layers; SwiGLU MLP)

Kept in BF16: SigLIP2 NaFlex vision encoder (model.vision_tower.*), LFM2 short-convolution operators (model.language_model.layers.*.conv.* — 22 conv layers), vision->LM projector (model.multi_modal_projector.*), token embeddings, lm_head, all RMSNorms (operator_norm / ffn_norm / q_layernorm / k_layernorm).

Runtime

Load with transformers>=5.0.0 (the lfm2_vl architecture is native — no trust_remote_code needed) using AutoModelForImageTextToText + AutoProcessor. The AWQ checkpoint is a compressed-tensors W4A16 (INT4, group size 128) model over the LFM2 language backbone; on-the-fly int4 dequant happens inside transformers.

What is and isn't quantized

This is a language-backbone-only AWQ. The SigLIP2 vision encoder, the 22 short-convolution operators of the hybrid LFM2 backbone, the vision->LM projector, and all norms/embeddings stay in BF16 — only the 8 full-attention layers' q/k/v/out projections and every layer's SwiGLU feed-forward (w1/w2/w3) are 4-bit. This keeps image understanding and the conv mixing path at full precision while shrinking the bulk of the LM weights.

Calibration

AWQ: 128 real image+text VQA samples streamed from HuggingFaceM4/the_cauldron (vqav2 config), each rendered through the model's own chat template + AutoProcessor so the LM backbone is calibrated on the true image-token activation distribution it sees at inference. Activation-aware scales use custom per-layer-anchored AWQ mappings (LFM2 uses operator_norm/ffn_norm rather than input_layernorm/post_attention_layernorm, and only the 8 full-attention layers carry q/k/v).

Prompt template & sampling

LFM2.5-VL uses a ChatML-like template. Load with transformers (native lfm2_vl, requires transformers>=5.0.0) via AutoModelForImageTextToText + AutoProcessor. Pass OpenAI-style messages with image content, e.g. messages=[{"role":"user","content":[{"type":"image","url":"..."},{"type":"text","text":"Describe this image."}]}], and render with processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", tokenize=True, return_dict=True). Do NOT insert <image> tags yourself — apply_chat_template inserts them automatically per image. Liquid recommends temperature=0.2, top_k=50, repetition_penalty=1.0 for text generation.

Recommended sampling: temperature=0.2, top_k=50, repetition_penalty=1.0 (per the Liquid AI model card); max_new_tokens per use case.

Verification

Validated end-to-end on NVIDIA Jetson Thor (Blackwell, aarch64, 128 GB unified memory), CUDA 13.0, PyTorch 2.13.0, transformers 5.15.1, compressed-tensors 0.17.0.

Loaded with (native lfm2_vl — no trust_remote_code; requires transformers>=5.0.0):

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

proc = AutoProcessor.from_pretrained("sahilchachra/LFM2.5-VL-3B-AWQ")
model = AutoModelForImageTextToText.from_pretrained(
    "sahilchachra/LFM2.5-VL-3B-AWQ", dtype=torch.bfloat16, device_map="cuda:0").eval()

Sanity check — real VQAv2 images, greedy decoding (do_sample=False):

Free-form ("Describe this image in detail."): “a small black and tan dog is seen riding a white surfboard on a calm, blue ocean…”

Decode speed: ~25.7 tok/s (single image, 128–256 new tokens, greedy) on one Thor GPU. Shapes verified: 122/122 quantized linears group-128 consistent; output logits (1, seq, 128000); SigLIP2 vision tower kept dense BF16.

Also verified serving under vLLM 0.27.1 (same answers, ~62 tok/s decode) — see Usage (vLLM) below.

Usage (vLLM)

Verified serving on vLLM 0.27.1 — the lfm2_vl architecture is native (no trust_remote_code) and the compressed-tensors W4A16 quantization is auto-detected. Requires a vLLM build with transformers>=5.0 (e.g. ≥ v0.27.1); older builds that pin transformers<5.0 cannot load the LFM2.5 tokenizer.

Offline (multimodal)

from vllm import LLM, SamplingParams

llm = LLM(
    model="sahilchachra/LFM2.5-VL-3B-AWQ",
    max_model_len=8192,                 # raise toward the base model's context as needed
    limit_mm_per_prompt={"image": 1},
)
messages = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://path/to/image.jpg"}},
    {"type": "text", "text": "Describe this image in detail."},
]}]
out = llm.chat(
    messages,
    SamplingParams(temperature=0.2, top_k=50, repetition_penalty=1.0, max_tokens=512),
)
print(out[0].outputs[0].text)

Server (OpenAI-compatible)

vllm serve sahilchachra/LFM2.5-VL-3B-AWQ \
    --max-model-len 8192 \
    --limit-mm-per-prompt '{"image": 1}'

Then send an image_url content part to /v1/chat/completions as usual.

Measured on one NVIDIA Jetson Thor GPU: loads in ~4 s (3.0 GiB weights), ~62 tok/s decode (single image, 128 new tokens, greedy).