Downloads · 30 days
14
18% of all-time downloads
ZoomFly/rwkv-vl-1.5v100m-nv_img_v3-260815
rwkv-vl-1.5v100m-nv_img_v3-260815 is a image-text-to-text model from ZoomFly. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Model lineage: this checkpoint was trained based on ZoomFly/rwkv-vl-1.5v100m-finevisionmax. It continues that model with the NV Image V3 reasoning-data stage; it is not a separately pretrained model.
Downloads · 30 days
14
18% of all-time downloads
All-time downloads
78
Public
Parameters
1.7B
3.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors3.3 GB · 100%
From the Hugging Face model README
Model lineage: this checkpoint was trained based on
ZoomFly/rwkv-vl-1.5v100m-finevisionmax. It continues that model with the NV Image V3 reasoning-data stage; it is not a separately pretrained model.
The ZoomFly/rwkv-vl-1.5v100m-finevisionmax is better for
This model is better for
Neither is sufficient for
This is a research checkpoint of RWKV-VL 1.5B-v100M continued on
vision-language data containing explicit <think>...</think> reasoning
traces. It supports two inference modes from the same weights:
<think>...</think> trace before its final answer.<think>\n</think>\n block. The model then answers without generating a
reasoning trace.The checkpoint demonstrates real gains from visual reasoning, especially on chart arithmetic, numerical calculation, and code reasoning. It does not reliably decide how much reasoning a question needs. Thinking can substantially hurt recognition, strict yes/no tasks, translation verification, and multi-image questions. Long traces can drift away from the requested task, repeat, or fail to close.
This model should therefore be used with caller-controlled mode routing. Direct mode is the safer default; enable thinking when a question genuinely requires several visual or computational steps.
| Property | Value |
|---|---|
| Architecture | Frozen Qwen-style vision encoder + visual MLP projector + RWKV7 language model |
| Parameters | 1,653,177,600 |
| Weight dtype | BF16 |
| Checkpoint size | Approximately 3.1 GB |
| RWKV hidden size / layers | 2,048 / 24 |
| Context length | 8,192 tokens |
| Vocabulary size | 65,536 |
| Image budget | 65,536 to 3,145,728 pixels per sample |
| Vision patch / merge size | 16 / 2 |
| EOS token IDs | 10060 (✿) and 0 (`< |
| Default behavior | Generate <think>...</think>, then answer |
| Direct-mode prefix | <think>\n</think>\n |
| Task type | Recommended mode | Observed behavior |
|---|---|---|
| Chart questions requiring lookup plus arithmetic | Thinking | ChartQA rises from 52.0% direct to 66.8% thinking on a 250-example evaluation |
| Numerical calculation | Thinking | MME accuracy rises from 52.5% to 85.0% |
| Code reasoning | Thinking | MME accuracy rises from 40.0% to 75.0% |
| Counting or position questions that require comparison | Usually thinking, with a strict token cap | MME count rises by 10 points and position by 15 points |
| Straight OCR | Either; thinking gain is small | 75.0% thinking versus 72.5% direct on 40 MME examples |
| Commonsense yes/no | Direct for efficiency | No measured accuracy gain from thinking |
| Object or scene existence | Direct | Thinking adds work and slightly reduces accuracy |
| Landmark, artwork, celebrity, scene, or poster recognition | Direct | Thinking causes hallucinated identification, task drift, and occasional repetition loops |
| Translation verification framed as yes/no | Direct | Thinking often performs a translation instead of answering yes/no |
| Simple visual lookup with a strict output format | Direct | Long reasoning may alter a correct first read or answer in the wrong format |
| Multi-image comparison, temporal frames, maps, or grounding | Direct, but treat this checkpoint as experimental | On the balanced multi-image diagnostic, direct scores 29.5% and thinking 10.5% |
These recommendations are empirical routing rules for this checkpoint, not a claim that reasoning is inherently harmful for recognition or multi-image tasks.
The repository contains custom model and processor code. Review the Python
files and use trust_remote_code=True only when you trust the repository.
The following versions were used to validate this export:
pip install "transformers==5.14.1" "accelerate==1.14.0" \
"flash-linear-attention==0.5.0" pillow
Install a PyTorch build appropriate for your CUDA environment separately.
from pathlib import Path
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "ZoomFly/rwkv-vl-1.5v100m-nv_img_v3-260815"
processor = AutoProcessor.from_pretrained(
model_id,
trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map="auto",
).eval()
def ask(
images: Image.Image | list[Image.Image],
question: str,
*,
thinking: bool,
max_new_tokens: int | None = None,
) -> str:
if isinstance(images, Image.Image):
images = [images]
images = [image.convert("RGB") for image in images]
content = [{"type": "image"} for _ in images]
content.append({"type": "text", "text": question})
messages = [{"role": "user", "content": content}]
prompt = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
if not thinking:
prompt += "<think>\n</think>\n"
inputs = processor(
text=[prompt],
images=images,
padding=False,
return_tensors="pt",
).to(model.device)
if max_new_tokens is None:
max_new_tokens = 512 if thinking else 128
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
)
generated_ids = output_ids[:, inputs.input_ids.shape[1] :]
return processor.batch_decode(
generated_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
image = Image.open(Path("example.jpg"))
# Use thinking for questions that require several visual/computational steps.
print(
ask(
image,
"Read the chart values, compute their median, and give the result.",
thinking=True,
max_new_tokens=768,
)
)
# Use direct mode for recognition, lookup, and strict-format questions.
print(
ask(
image,
"Is there a red car in the image? Answer only Yes or No.",
thinking=False,
)
)
The direct-mode prefix is part of the input, so the decoded continuation normally contains only the answer. Direct mode avoids generating a trace, but it does not restore the weights of the model before thinking training.
Pass all images in order and include one image item per image in the message:
images = [
Image.open("frame_1.jpg"),
Image.open("frame_2.jpg"),
Image.open("frame_3.jpg"),
]
answer = ask(
images,
"Which frame contains the white cabinet? Answer with 1, 2, or 3.",
thinking=False,
)
print(answer)
The processor correctly preserves the number and order of image feature blocks. However, this checkpoint may skip an image, merge evidence across frames, or copy one description to several images during joint reasoning.
Evaluations used lmms-eval commit v0.6-167-g6619ef61, greedy decoding,
and the checkpoint's custom batch-invariant generation path. Reasoning tags
were removed before task scoring.
Two modes were evaluated:
<think>\n</think>\n, with up
to 512 tokens. This was evaluated through a companion export with the prefix
embedded in its chat template; the weights were identical.This is a deterministic evaluation of the first 250 ChartQA test examples, not the complete benchmark.
| Model | Thinking | Direct | Thinking minus direct |
|---|---|---|---|
| This checkpoint | 66.8% | 52.0% | +14.8 points |
| FineVisionMax comparison checkpoint | 55.2% | 53.2% | +2.0 points |
For this checkpoint:
</think> and hit the 2,048-token
limit.Thinking helps when the model must read several values and then calculate or compare them. Thinking hurts when it selectively drops one of the values it already identified, misreads a chart value during a long trace, makes an arithmetic error, or reinterprets the question.
The following table reports per-question accuracy over all 2,374 MME questions. It is useful for comparing modes, but it is different from MME's official category-aggregated point score.
| MME subtask | Samples | Thinking | Direct | Difference |
|---|---|---|---|---|
| Text translation verification | 40 | 22.50% | 72.50% | -50.00 points |
| Landmark recognition | 400 | 66.25% | 83.50% | -17.25 points |
| Artwork recognition | 400 | 58.75% | 75.00% | -16.25 points |
| Celebrity recognition | 340 | 62.35% | 75.59% | -13.24 points |
| Scene recognition | 400 | 80.00% | 85.75% | -5.75 points |
| Poster recognition | 294 | 63.95% | 68.71% | -4.76 points |
| Existence | 60 | 93.33% | 96.67% | -3.34 points |
| Commonsense reasoning | 140 | 70.71% | 70.71% | 0.00 points |
| OCR | 40 | 75.00% | 72.50% | +2.50 points |
| Color | 60 | 85.00% | 76.67% | +8.33 points |
| Count | 60 | 80.00% | 70.00% | +10.00 points |
| Position | 60 | 85.00% | 70.00% | +15.00 points |
| Numerical calculation | 40 | 85.00% | 52.50% | +32.50 points |
| Code reasoning | 40 | 75.00% | 40.00% | +35.00 points |
| Concrete group | 2,074 | 68.76% | 78.30% | -9.54 points |
| Abstract group | 300 | 67.33% | 64.67% | +2.66 points |
| All questions | 2,374 | 68.58% | 76.58% | -8.00 points |
The official MME point totals tell a complementary story because they weight and aggregate task categories differently:
| Mode | Perception points | Cognition points | Total points |
|---|---|---|---|
| Thinking | 1,317.67 | 441.07 | 1,758.74 |
| Direct | 1,361.77 | 351.43 | 1,713.20 |
Thinking improves the official cognition component enough to raise the total point score, while direct mode answers more individual questions correctly. Report the metric definition whenever citing either result.
This is a paired, deterministic 200-example subset of the first official MuirBench parquet shard: 40 examples from each of five tasks, 2-9 images per example, seed 42. It is not the official full MuirBench score.
| Task | Direct | Thinking |
|---|---|---|
| Action Understanding | 32.5% | 10.0% |
| Counting | 25.0% | 10.0% |
| Geographic Understanding | 22.5% | 2.5% |
| Image-Text Matching | 42.5% | 20.0% |
| Visual Grounding | 25.0% | 10.0% |
| Overall | 29.5% | 10.5% |
The FineVisionMax comparison checkpoint reaches 37.0% semantic accuracy on
the same subset. Its strict task score is only 3.5% because it often begins
with The correct answer is B... instead of emitting a bare option letter;
the 37.0% comparison extracts explicit answer statements while leaving
unextractable responses wrong.
In thinking mode, only 79/200 traces close within 1,024 tokens. The remaining 121 hit the token limit without emitting a final answer. Of the 79 completed traces, 21 are correct. Increasing the reasoning budget increases the number of completed thoughts, but does not recover direct-mode accuracy.
All tested image blocks are present and causally affect recurrent state. The failure is not caused by the processor dropping or reordering images. The model instead has difficulty retaining distinct image identities and binding visual evidence to the requested answer during a long recurrent trace.
Thinking is most useful when a direct answer would otherwise be a guess and the trace can supply a checkable intermediate computation:
Productive traces explicitly identify the relevant evidence, perform a short
calculation or comparison, close </think>, and then commit to one answer.
Thinking is usually unnecessary or actively harmful when the answer depends on one stable visual judgment:
Observed failure patterns include:
Wait, Let's re-evaluate, or
literal phrase/token loops prevent </think> closure.Controlled component swaps found that this checkpoint still recognizes simple visual attributes, but may fail to bind them to an answer choice:
The regression is reproduced by swapping only the trained RWKV recurrent stack into the comparison model; swapping only the projector or LM head does not reproduce it. This means direct mode avoids the generated trace but does not remove all effects of thinking training on the recurrent model.
Generation also showed residual batch-composition sensitivity in some greedy
replays. The reported lmms-eval runs used the custom batch_invariant=true
path, but some stopping outcomes still changed when batch companions or device
ranks changed. For critical use, generate one sample at a time and verify the
answer independently.
Flash Linear Attention warns that its RWKV implementation may differ from the official RWKV implementation. Results in this card use one consistent runtime for all paired comparisons, but absolute behavior may change with another kernel or implementation.
RWKV-VL does not expose a Transformer-style text-to-image attention matrix,
and output_attentions=True is unsupported. Visual information is inserted as
projected image-token embeddings and propagated through recurrent state.
</think> as a failed generation instead of assuming
that a larger budget will eventually produce a correct answer.The correct answer is B instead of accepting only a leading letter.The available training log records:
ZoomFly/rwkv-vl-1.5v100m-finevisionmax and continued with the NV Image V3
reasoning-data stage;Nemotron-Image-Training-v3-parquet as the image-text source;3e-5;3e-5;The run processed approximately 872,960 examples before the data source was exhausted. The public source mixture is dominated by single-image examples; its sampled multi-image portion is primarily multi-page document data rather than independent-scene comparison or temporal-frame reasoning. This may not exactly match the post-filtered training snapshot.
Appropriate uses include:
This checkpoint is not recommended for autonomous high-stakes decisions, unverified factual identification, safety-critical visual inspection, or applications that require stable hidden reasoning or guaranteed multi-image coverage.
No license is declared in this model card. Before public release, the publisher should verify and add the licenses and attribution requirements for the base model, RWKV/FLA components, vision encoder, training data, and bundled custom code.