qwen2vl-polygen-lora-r16-1000

Polygen LoRA for Qwen2-VL-2B (r=16, 1000 steps, ScienceQA)

A LoRA adapter for Qwen/Qwen2-VL-2B-Instruct that is part of Datasent's polygen visual-token compression recipe. It teaches the LLM to interpret a compressed visual-token sequence produced by the polygen splice between Qwen2-VL's vision encoder and its language model. Paired with the splice, it recovers baseline accuracy under aggressive compression; on a 7B backbone the same recipe exceeds the un-spliced baseline.

Interactive side-by-side demo: datas3nt/qwen2vl-polygen

What this adapter is (and isn't)

This is not a general-purpose Qwen2-VL fine-tune. It is one half of a two-part configuration:

1. The polygen splice sits between Qwen2-VL's merger output and its LLM input and replaces the per-image visual-token sequence with a much shorter structured representation (or, in the hook splice path, an in-place reconstruction that preserves the original token count while compressing the information).
2. This LoRA adapter teaches the LLM to interpret that compressed representation. Without the adapter, the LLM treats the compressed sequence as out-of-distribution noise; with the adapter, accuracy returns to baseline (or above it on larger backbones).

The adapter alone (no polygen splice) gives a small accuracy lift on ScienceQA-distribution-shifted tasks via cross-benchmark transfer, but its purpose is the joint configuration with the splice.

Recipe

Results

The adapter is evaluated as the LoRA half of Datasent's hybrid LOSSLESS / COARSE polygen splice, which exposes five operating tiers (T=0 most aggressive compression through T=4 lossless storage). Below are cross-benchmark numbers at T=4 + LoRA (the conservative, lossless-storage tier), evaluated against an un-spliced un-LoRA baseline.

Compute and storage are reported as attention-FLOP reduction (a FLOP count, not wall-clock) and storage compression, both a function of the operating tier. On the 7B backbone: ~71% fewer attention FLOPs at T=2 (accuracy +7.4pp) and 5.1x storage at T=3 (accuracy +8.4pp). The 7B card and the live demo carry the full per-tier curve.

The LoRA was trained only on ScienceQA. The lifts on MMMU, VQAv2, and DocVQA are pure cross-benchmark transfer; the adapter never saw any of those distributions during training. The DocVQA hook-path result uses a length-preserving reconstruction that preserves Qwen2-VL's 2-D positional encoding for high-token-count document images; the GPU-resident reconstruction in the SDK removes the hook path's earlier per-forward compute overhead.

How to use

This adapter is intended to be loaded by Datasent's polygen SDK as part of a two-step setup:

from polygen.integrations.qwen2vl import PolygenQwen2VL

model = PolygenQwen2VL.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct",
    lora_adapter="datas3nt/qwen2vl-polygen-lora-r16-1000",
    operating_tier="T=4",
    splice_mode="auto",          # bypass for moderate L_v, hook for high L_v
    torch_dtype="float16",
    device_map="cuda",
)

For evaluation without the polygen splice (rare; the adapter's value is in the joint configuration), it can be loaded as a standard PEFT adapter:

from peft import PeftModel
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor

base = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct",
    torch_dtype="float16",
    device_map="cuda",
)
model = PeftModel.from_pretrained(base, "datas3nt/qwen2vl-polygen-lora-r16-1000")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

Intended use

• Primary: evaluation and demonstration of Datasent's polygen visual-token compression on Qwen2-VL-2B, paired with the splice.
• Secondary: cross-benchmark transfer studies on ScienceQA-adjacent vision-language tasks (the +1.90pp transfer effect on VQAv2 is the cleanest case).
• Not intended for: general-purpose Qwen2-VL fine-tuning, production deployment without the polygen SDK, or stand-alone use on tasks that diverge significantly from ScienceQA's distribution.

Limitations

• 2B-only. This adapter targets Qwen2-VL-2B-Instruct. The same recipe on a 7B backbone (with separately-trained 7B LoRA) is what produces the +8.4pp accuracy / ~71% fewer-attention-FLOPs headline; that artifact lives under a separate repository.
• Recipe assumes the polygen splice. Without the visual-token compression the adapter underperforms a full fine-tune; its value is in the joint configuration.
• English training data. ScienceQA is an English-language benchmark; behaviour on non-English captions or non-Latin scripts is undocumented.
• n=300 train sample. The adapter is intentionally small and cheap to train (~30 minutes on a single H200). Performance on out-of-distribution tasks beyond the four benchmarks above has not been characterized.

Citation

@misc{datasent-polygen-vlm-2026,
  title  = {Polygen: visual-token compression for vision-language models},
  author = {Datasent, Inc.},
  year   = {2026},
  url    = {https://huggingface.co/spaces/datas3nt/qwen2vl-polygen},
}

License

Distributed under a Datasent evaluation license. Contact [email protected] for production licensing.