llm-jp-4-vl-9b

<div align="center" style="line-height: 1;"> <h1>LLM-jp-4-VL 9B</h1>

| <a href="https://llm-jp.nii.ac.jp/blog/llm-jp-4-vl-9b/" target="_blank">📄 Blog</a>  | <a href="https://github.com/llm-jp/llm-jp-4-vl" target="_blank">🧑‍💻 Code</a>  |

<br/> </div>

LLM-jp-4-VL 9B is a vision-language model developed by LLM-jp.

It is built upon llm-jp/llm-jp-4-8b-thinking and SigLIP 2 So400m.

Compared with the earlier LLM-jp-4-VL 9B beta, the new model is trained on datasets that have no license or terms-of-use issues for commercial use. It is also enhanced to generate a reasoning trace before a final answer. Our evaluation shows improved performance over the beta version, particularly on text-only tasks and challenging chart-understanding benchmarks such as CharXiv and HakushoBench.

<img src="./benchmark_comparison_hatched.png" width="80%"/>

To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the survey form.

Model Architecture

The model architecture is inspired by InternVL3.0. It consists of a language model, a vision encoder, and a lightweight projector.

• LLM: llm-jp/llm-jp-4-8b-thinking (8.6B)
• Vision Encoder: google/siglip2-so400m-patch16-512 (0.4B)
• Projector: 2-layer MLP

Chat Template

We adopt a customized chat template based on an extension of the OpenAI Harmony format.

In this template, a sequence of image tokens is wrapped with special tokens <|image_start|> and <|image_end|>, and inserted into the user message. Because this is a reasoning model, the assistant may first produce an analysis message (reasoning content) and then a final message (the answer). An example input/output is shown below:

<|start|>system<|message|>{system_message}<|end|>
<|start|>user<|message|><|image_start|><|image_pad|>...<|image_pad|><|image_end|>What is the animal shown in this image?<|end|>
<|start|>assistant<|channel|>analysis<|message|>The image shows a small four-legged animal with pointed ears and whiskers ...<|end|><|start|>assistant<|channel|>final<|message|>It is a cat.<|return|>

Training Data

The training data is a mixture of four large-scale datasets (≈29.3M samples in total):

• Jagle (9.2M): a Japanese multimodal dataset built by LLM-jp.
• RefinedVision (12.0M): a refined version of FineVision built by LLM-jp. Of the 185 subsets in FineVision, we removed 61 that include images with licensing concerns and poor quality. Of the remaining 124 subsets, we regenerated QA pairs for 58 subsets whose original QA pairs were of low quality or had license/terms-of-use concerns (e.g., QA generated by GPT-4o or other proprietary models): keeping the original images, we regenerated the QA from scratch with Qwen3.5 397B-A17B using per-category generation prompts. The other 66 subsets were used as-is. See the RefinedVision dataset page for details.
• Nemotron-Image-Training-v3 subset (5.0M): an English multimodal dataset built by NVIDIA, from which we removed subsets with license/terms-of-use issues. The exact list of subsets we used is given in nemotron_subsets.md.
• llm-jp-4-thinking-sft-data (3.2M): a text-only Japanese reasoning SFT dataset built by LLM-jp.

Training Details

The language model and vision encoder are initialized from their pretrained weights, while the projector is initialized randomly. We then train the model in a single stage for 120,000 steps, with all three components trainable from the start (no projector-only warmup or frozen-backbone stage).

We use different maximum learning rates for each component: 2e-5 for both the language model and the vision encoder, and 1e-4 for the projector.

The learning rate follows the Warmup–Stable–Decay (WSD) schedule, with a warmup phase of 2,000 steps, followed by stable training, and linear decay starting after 80% of the total training steps.

Evaluation

We evaluate our model and all baselines with simple-evals-mm, our evaluation framework for VLMs, across a broad suite of English and Japanese multimodal and text-only benchmarks. For several Japanese benchmarks, we use the refined versions from the JAMMEval collection.

Evaluation settings

LLM-jp-4-VL 9B is decoded greedily (temperature=0.0) with max_new_tokens=32k and reasoning effort medium. Each baseline uses its own recommended sampling temperature; for the reasoning models Qwen3.5-9B and Gemma-4-12B, we increase max_new_tokens to 64k to avoid truncating long chains of thought. Scores are averaged over 3 runs.

Per-benchmark score vs training step

As training progresses, the model improves on most multimodal benchmarks. On the text-only benchmarks, it preserves the reasoning ability of the base LLM (llm-jp-4-8b-thinking), so adding vision capability does not degrade its language reasoning.

Per-benchmark results

Domain-level averages

The following table reports the average score for each domain, along with the overall average (All), the average over Japanese benchmarks (Ja-Avg), and the average over English benchmarks (En-Avg). Benchmarks with fewer than 100 examples are excluded from these averages.

Compared to the beta version, LLM-jp-4-VL 9B showed improved performance on text-only tasks such as GPQA, as well as on challenging chart understanding tasks such as CharXiv Reasoning and HakushoBench. It also outperformed Gemma-4-12B on some tasks.

Usage

Install requirements:

uv add "torch==2.8.0" "transformers==4.57.0" "flash-attn==2.8.3" "pillow==11.3.0"

Below is the sample code to run the model.

import torch
import requests
from PIL import Image
from transformers import AutoProcessor, AutoModel

model_id = "llm-jp/llm-jp-4-vl-9b"

# load model
model = (
    AutoModel.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16,
        trust_remote_code=True,
        use_flash_attn=True,
    )
    .eval()
    .cuda()
)

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Example images are fetched from the GitHub repo so this snippet runs anywhere.
ASSETS = "https://raw.githubusercontent.com/llm-jp/llm-jp-4-vl/main/assets"


def load_image(name):
    return Image.open(requests.get(f"{ASSETS}/{name}", stream=True).raw).convert("RGB")


def generate(messages, max_new_tokens=1024, temperature=0.0, reasoning_effort="medium"):
    inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_dict=True,
        return_tensors="pt",
        reasoning_effort=reasoning_effort,  # "low" (direct) / "medium" / "high"
    ).to(model.device)

    if "pixel_values" in inputs:
        inputs["pixel_values"] = inputs["pixel_values"].to(dtype=model.dtype)

    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=temperature > 0,
        temperature=temperature if temperature > 0 else None,
    )

    # The generated turn is `<|channel|>analysis<|message|>{thinking}<|end|>` (the
    # reasoning) followed by `<|channel|>final<|message|>{answer}<|return|>`.
    # Return (thinking, answer). decode() can put spaces around the harmony
    # markers, so we match them space-tolerantly.
    import re

    text = processor.decode(outputs[0], skip_special_tokens=False)
    m = re.search(r"<\|channel\|>\s*analysis\s*<\|message\|>(.*?)<\|end\|>", text, re.S)
    thinking = m.group(1).strip() if m else ""
    finals = list(re.finditer(r"<\|channel\|>\s*final\s*<\|message\|>", text))
    answer = text[finals[-1].end():] if finals else text
    answer = re.sub(r"<\|[^|]*\|>", "", answer).replace(processor.tokenizer.eos_token, "").strip()
    return thinking, answer


def show(messages, **kwargs):
    thinking, answer = generate(messages, **kwargs)
    print(f"[thinking] {thinking}")
    print(f"[answer]   {answer}\n")


# -----------------------
# 1. Text-only
# -----------------------
messages = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "富士山について簡潔に説明してください。"}],
    }
]
show(messages)
# [thinking] User asks to briefly explain Mt. Fuji; list key facts (location, height, ...).
# [answer]   **富士山(ふじさん)** — 静岡・山梨県境の活火山、標高3,776 m(日本最高峰)。
#            成層火山で、2013年に世界遺産に登録。登山シーズンは7〜9月上旬。

# -----------------------
# 2. Single image
# -----------------------
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": load_image("kaonashi.jpg")},
            {"type": "text", "text": "このキャラクターの名前は何ですか？"},
        ],
    }
]
show(messages)
# [thinking]
# [answer]   カオナシ

# -----------------------
# 3. Multi-image
# -----------------------
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": load_image("Shiba_inu.jpg")},
            {"type": "image", "image": load_image("yesoensis.jpg")},
            {"type": "text", "text": "それぞれの動物の名前を教えてください。"},
        ],
    }
]
show(messages)
# [thinking] 画像には2匹の動物が写っている。1匹は犬(柴犬)、もう1匹は鹿。
# [answer]   柴犬と鹿です。

# -----------------------
# 4. Multi-turn (image + follow-up)
# -----------------------
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": load_image("kaonashi.jpg")},
            {
                "type": "text",
                "text": "このキャラクターが登場する映画のタイトルは何ですか？",
            },
        ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "千と千尋の神隠し"}],
    },
    {
        "role": "user",
        "content": [{"type": "text", "text": "監督は誰ですか？"}],
    },
]
show(messages)  # reasoning_effort="medium" (recommended for vision)
# [answer]   宮崎駿

# -----------------------
# 5. Reasoning (text-only; effort is controllable)
# -----------------------
messages = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "9.11 と 9.9 はどちらが大きいですか？理由も述べてください。"}],
    }
]
show(messages, reasoning_effort="high")
# [thinking] Compare 9.11 vs 9.9: integer parts equal (9); 0.9 > 0.11, so 9.9 is larger.
# [answer]   **答え: 9.9 の方が大きいです。** 整数部は同じ 9、小数部は 0.9 > 0.11 なので 9.9 > 9.11。

For inputs that include images, the reasoning effort is not yet reliably controllable. We therefore recommend reasoning_effort="medium" for vision inputs.

For more details, please refer to the official GitHub repository: https://github.com/llm-jp/llm-jp-4-vl

LICENSE

Apache License 2.0

Citation

If you find our work useful, please consider citing the following papers:

@misc{sugiura2026jaglebuildinglargescalejapanese,
      title={Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models},
      author={Issa Sugiura and Keito Sasagawa and Keisuke Nakao and Koki Maeda and Ziqi Yin and Zhishen Yang and Shuhei Kurita and Yusuke Oda and Ryoko Tokuhisa and Daisuke Kawahara and Naoaki Okazaki},
      year={2026},
      eprint={2604.02048},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.02048},
}

@misc{sugiura2026jammevalrefinedcollectionjapanese,
      title={JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation},
      author={Issa Sugiura and Koki Maeda and Shuhei Kurita and Yusuke Oda and Daisuke Kawahara and Naoaki Okazaki},
      year={2026},
      eprint={2604.00909},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.00909},
}