SigLIP-so400m-patch14-384

SigLIP-SO400M-patch14-384 - Renesas X5H

Introduction

This repository contains the SigLIP-SO400M-patch14-384 dual-encoder model, optimized for the Renesas X5H platform for zero-shot image-text similarity inference.

SigLIP is a dual-encoder model consisting of a vision encoder and a text encoder. It is not a generative model -- it does not produce text output. Instead, it computes similarity scores between images and text labels, enabling zero-shot image classification and image-text matching.

• Model Architecture: SigLIP uses a Vision Transformer (ViT) for image encoding and a Transformer for text encoding. Both encoders produce L2-normalized embeddings that are compared via dot-product similarity with a learned scale and bias, followed by sigmoid activation.
• Model Summary:

  Parameter	SigLIP-SO400M-patch14-384
  HIDDEN_SIZE	1152
  INTERMEDIATE_SIZE	4304
  NUM_HEADS	16
  HEAD_DIM	72
  NUM_LAYERS	27 (both encoders)
  PATCH_SIZE	14
  IMAGE_SIZE	384
  NUM_PATCHES	729
  VOCAB_SIZE	32000
  MAX_TEXT_SEQ_LEN	64

• Source Model: google/siglip-so400m-patch14-384

Performance

The following performance metrics were measured on the Renesas X5H board.

Prerequisites

To run the model, you need:

1. Renesas X5H Board with SDK v4.32.0 or v4.34.0
2. Hugging Face CLI: For downloading the model and installer.

Board Setup

Power Cycle

The X5H board must be power-cycled via USB serial before first use or after any NPU hang.

echo "POWER#OF" > /dev/ttyUSB2
sleep 3
echo "POWER#ON" > /dev/ttyUSB2

Wait for the board to boot (typically 30-60 seconds).

NPU Setup

Run setup_npu.sh exactly once after each power cycle. Do not run it multiple times without rebooting first.

bash ./setup_npu.sh npu0

This loads kernel modules and starts all 14 NPU firmware cores.

Important: FP16 and W4A16 runners cannot be used back-to-back on the same boot. Running W4A16 corrupts NPU internal state, causing subsequent FP16 runs to produce NaN. Power cycle the board when switching between variants.

Deployment

SigLIP-SO400M-patch14-384 (FP16)

Two versions are available:

• xOS-v4.34: VE encoding

1. Download the installer siglip-runner-3.0.0-Linux.sh from Files and versions tab under fp16/binaries/rcar-x5hv1/xOS-v4.34/ (optimized) or fp16/binaries/rcar-x5hv1/xOS-v4.32/ (baseline).
2. Copy the installer to the X5H board and run the installer.

   bash ./siglip-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   

3. Expected directory structure on the X5H board.

    siglip-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-runner
    └── setup_npu.sh
   

Inference - SigLIP-SO400M-patch14-384 (FP16)

bash ./setup_npu.sh npu0
./siglip-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -s

Expected output:

similarity_score: 6.5136025660e-03 (0.0065136026)

SigLIP-SO400M-patch14-384 (W4A16)

1. Download the installer siglip-w4a16-runner-3.0.0-Linux.sh from Files and versions tab under w4a16/binaries/rcar-x5hv1/xOS-v4.34/ folder.
2. Copy the installer to the X5H board and run the installer.

   bash ./siglip-w4a16-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   

3. Expected directory structure on the X5H board.

    siglip-w4a16-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── graphs
    │   ├── siglip-ve-w4a16
    │   └── siglip-te-w4a16
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-w4a16-runner
    └── setup_npu.sh
   

Inference - SigLIP-SO400M-patch14-384 (W4A16)

bash ./setup_npu.sh npu0
./siglip-w4a16-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -g graphs -s

Expected output:

similarity_score: 5.3571168333e-02 (0.0535711683)

CLI Reference

Usage: siglip-runner -m <gguf> (-i <image> | -L <image_list>) (-t <token_ids> | -f <prompts_file>) [options]
  -m  Path to SigLIP GGUF model
  -i  Path to image (JPEG, PNG, BMP, or PPM)
  -L  File with one image path per line (batch image mode)
  -t  Comma-separated token IDs (single prompt mode)
  -f  Prompts file for batch mode (label|token_ids per line)
  -d  NPU device path (default: /dev/snps/arcnet0/app0)
  -s  Print performance metrics

W4A16 adds:
  -g  Graph directory (contains siglip-ve-w4a16/ and siglip-te-w4a16/)

Token IDs are SentencePiece encoded. The runner pads to 64 tokens internally and places EOS (token ID 1) at position 63.