laguna-s-2.1-vision

Laguna-S-2.1 Vision

Laguna-S-2.1 with sight. A vision-language model that attaches the Qwen3-VL vision tower to poolside's Laguna-S-2.1 through a trained projector.

Laguna-S-2.1 is a strong code and reasoning model with no image input. This checkpoint adds it without touching a single Laguna weight. Both the text backbone and the vision tower are frozen and byte-identical to their upstream releases. The only newly trained parameters are the 35.4M projector that maps Qwen3-VL's 1152-dim patch embeddings into Laguna's 3072-dim token space.

Code: https://github.com/numinousmuses/laguna-s-2.1-vision — training, evaluation with the blind control, the projector module, and full inference code.

Results

Every benchmark ran twice: once normally, once with the image features replaced by Gaussian noise of the same shape and scale. The gap between the two is the measurement. 300 questions per benchmark, per arm.

Read the chance column before the sighted column.

• HallusionBench is yes/no, so its floor is 50%. A sighted score of 51.67% is at chance. The 47-point gap says the model is using the image; the absolute score says it is not yet resisting the trap questions. See next steps.
• TextVQA and OCRVQA are open-ended and scored here by normalised containment, which is looser than official VQA accuracy. Treat those two rows as internal measurements, not leaderboard-comparable numbers.
• Blind scores below chance are expected. Noise in the image slots is worse than no image at all, because the model still attends to the slots.
• An earlier checkpoint scored higher. Step 900, one epoch, measured 41.0% sighted and 13.67% blind on MMMU-Pro, gap 27.33. It was deleted by a retention rule before assembly, so these weights are step 2070. See limitations.

The blind arm is the point of the table. Sighted accuracy on its own cannot separate a model reading images from a language model answering off the question wording. In our own experiments a different graft scored 41.67% sighted on MMMU-Pro and 40.0% blind, a gap of 1.67 points. It was not using the images at all, and its headline number was indistinguishable from this model's.

Limitations

• This is not a natively trained VLM. The language model never saw an image during its own training. It receives features translated into its embedding space by a small MLP. Models trained end to end on multimodal data score considerably higher on the same benchmarks.
• The released checkpoint is supervised finetuning only. No reinforcement learning stage is included in these weights.
• Trained on short answers. The training mixture is short-form visual QA with captions excluded, because long targets are off-policy for a frozen backbone and carry high loss regardless of how well the features are aligned. Long-form description and free-form generation about images were never trained and should be expected to be weak.
• Partial warm start. The Qwen3-VL merger outputs 4096 and Laguna's hidden size is 3072, so linear_fc2 was randomly initialised. Only 4 of 6 projector tensors transferred.
• Hallucination is the clearest weakness. At-chance HallusionBench means the model reads the image but does not reliably contradict a false premise in the question.
• Fine text is limited by image token budget. Patches are merged 2x2 before the projector, so dense documents and small print lose resolution before the language model ever sees them.
• These are the second-epoch weights, and the first epoch was better. At step 900 (57,600 samples) training loss was 1.30 and MMMU-Pro was 41.0% sighted, gap 27.33. At step 2070 (132,480 samples) loss was 0.61 and MMMU-Pro was 38.33%, gap 25.33. A keep-last-3 retention rule deleted step 900 before the artifact was assembled, so the released projector is step 2070. Every number in the table above was measured on these exact weights.
• That 2.67-point difference is not statistically established. At p near 0.40 and n=300, one arm carries a standard error of 2.83 points and an unpaired difference of two arms carries 4.0 points, so the observed gap sits inside one standard error of zero. Both arms ran the same 300 questions and a paired test would have smaller variance, but the per-item outputs and every checkpoint were deleted with the training volume, so it cannot be run. Treat "one epoch was better" as unmeasured rather than as a small measured effect. What the run does show is that training loss halved while held-out accuracy did not improve, so loss is not tracking the thing you care about.

Next steps

Reinforcement learning, at a scale that can be measured. GRPO on the projector with verifiable rewards is the obvious next stage, and it is cheap because only 35.4M parameters move. Our own 10-step probe is not evidence either way: step-to-step reward variance was std 0.024 on a mean of 0.122, wider than anything 10 steps could produce. A real run needs hundreds of steps before the benchmark suite can settle it.

Unfreeze the language model. Everything here holds Laguna byte-identical, which is what makes the text behaviour guarantee below true. Relaxing that, even to LoRA on attention projections only, is the largest available gain and the point at which the model stops being a drop-in replacement for the base weights.

Raise the image token budget. Dropping the 2x2 merge quadruples image tokens and should move TextVQA and OCRVQA most. It costs sequence length, so it trades against throughput rather than against quality.

Train against hallucination directly. HallusionBench sits at its 50% floor. That needs contrastive data where the question's premise is false, which the current short-answer mixture contains almost none of.

More alignment data. 57,600 samples is small. The gap was still moving between checkpoints when we stopped, which means the curve had not flattened.

What this would not fix. No amount of projector training makes a frozen text model reason about images the way an end-to-end multimodal model does. The projector translates; it does not teach. The ceiling here is set by what Laguna can already do once the content arrives in its embedding space.

Text behaviour is unchanged, structurally

With no image tokens in the prompt, the forward pass is bit-identical to stock Laguna-S-2.1. Every backbone weight is frozen, the projector is not in the text path, and the added placeholder id never appears. This is a property of the construction, not a benchmark result.

What is in this repository

mm_projector.safetensors      35.40 M   the only trained part
vision_tower.safetensors     576.39 M   Qwen3-VL tower, frozen
config.json                             wires backbone, tower, projector
preprocessor_config.json                Qwen preprocessing, patch 16
tokenizer.json, tokenizer_config.json, generation_config.json

The text shards are not included. They are byte-identical to poolside/Laguna-S-2.1, so point --model-path at the upstream checkpoint and load these files alongside it. Copying 219 GB would buy nothing.

Quickstart

import torch, json
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoImageProcessor
from transformers.models.qwen3_vl_moe.modeling_qwen3_vl_moe import Qwen3VLMoeVisionModel
from transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe import Qwen3VLMoeVisionConfig
from safetensors.torch import load_file

REPO = "<this repo>"
BASE = "poolside/Laguna-S-2.1"
IMAGE_TOKEN_ID = 100351

lm  = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16,
                                           device_map="auto", attn_implementation="sdpa")
tok = AutoTokenizer.from_pretrained(BASE)

cfg   = json.load(open(f"{REPO}/config.json"))
tower = Qwen3VLMoeVisionModel(Qwen3VLMoeVisionConfig(**cfg["vision_config"]))
tower.load_state_dict({k[len("model.visual."):]: v
                       for k, v in load_file(f"{REPO}/vision_tower.safetensors").items()},
                      strict=False)
tower = tower.to("cuda", torch.bfloat16)

proc = AutoImageProcessor.from_pretrained(REPO)
# projector: see modeling code at https://github.com/numinousmuses/laguna-s-2.1-vision

Full inference code, including the projector module and the blind-control evaluation harness, is in the linked GitHub repository.

License

MIT, following both parents. The projector weights are released under MIT. Redistributed upstream weights keep their original terms.

Acknowledgements

Built on poolside's Laguna-S-2.1 and Alibaba's Qwen3-VL. Neither team was involved in this work. Please do not send issues about this checkpoint to them.