UniICL

UniICL

<p align="center"> <a href="https://arxiv.org/abs/2603.24690"> <img src="https://img.shields.io/badge/arXiv-2603.24690-b31b1b?style=flat" alt="arXiv"> </a> <a href="https://github.com/xuyicheng-zju/UniICL"> <img src="https://img.shields.io/badge/Code-GitHub-2f80ed?style=flat" alt="Code"> </a> <a href="https://huggingface.co/datasets/xuyicheng-zju/UniICL-760K"> <img src="https://img.shields.io/badge/Hugging%20Face-Dataset-f9d849?style=flat" alt="Dataset"> </a> </p>

This repository hosts the released UniICL model assets. UniICL is a unified multimodal in-context learning framework for both visual understanding and generation. The accompanying code, benchmark, and dataset utilities are available at GitHub, and the training data release is available at UniICL-760K.

Expected Layout

The open-source code assumes this root structure:

UniICL/
  UniICL-Bench/
  UniICL-760K/
  UniICL/
  local_paths.py

Before conversion, training, or evaluation, edit local_paths.py in the code repository:

UNIICL_BASE_MODEL = "/path/to/base-model"
UNIICL_FINETUNED_MODEL = "/path/to/finetuned-checkpoint"
UNIICL_TARGET_CHECKPOINT = "/path/to/finetuned-checkpoint"

UNIICL_BASE_MODEL should point to the released base model assets. UNIICL_FINETUNED_MODEL is used by evaluation and inference. UNIICL_TARGET_CHECKPOINT is the directory prepared for evaluation after training.

Checkpoint Contents

A CAPM-enabled checkpoint should contain:

checkpoint/
  ema.safetensors or model.safetensors
  capm_config.json
  vocab.json
  merges.txt
  tokenizer.json
  tokenizer_config.json
  llm_config.json
  vit_config.json
  ae.safetensors

For compatibility with older internal checkpoints, the public loader also accepts prism_config.json and automatically remaps legacy prism.* checkpoint keys to the public capm.* module names.

Training

First convert the UniICL-760K public train annotations:

cd UniICL-760K
bash convert_unified.sh

Train the unified ICL model without CAPM:

cd ../UniICL
NPROC_PER_NODE=8 TOTAL_STEPS=10000 bash scripts/train_uniicl.sh

Train with CAPM enabled:

cd ../UniICL
NPROC_PER_NODE=8 TOTAL_STEPS=10000 bash scripts/train_uniicl_capm.sh

After training, copy tokenizer, config, and autoencoder assets from the base model into the evaluation checkpoint directory:

bash scripts/prepare_uniicl_checkpoint.sh

Inference

Understanding example:

python scripts/run_uniicl_inference.py \
  --model-path /path/to/checkpoint \
  --image /path/to/example.jpg \
  --prompt "Answer the question about this image." \
  --understanding-output

Generation example:

python scripts/run_uniicl_inference.py \
  --model-path /path/to/checkpoint \
  --prompt "Generate an image of a lighthouse at dusk." \
  --output-image generated.png

Use --no-capm to disable CAPM at inference time for ablations.

Citation

@article{xu2026uniicl,
  title={UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy},
  author={Xu, Yicheng and Zhang, Jiangning and Xue, Zhucun and Hu, Teng and Yi, Ran and Hu, Xiaobin and Liu, Yong and Tao, Dacheng},
  journal={arXiv preprint arXiv:2603.24690},
  year={2026}
}