orena-focus-frame-qwen3.6-27b-lora

ORena FOCUS FRAME — Qwen3.6-27B LoRA adapter

LoRA adapter submitted to the FRAME track of the ORena SAVE FOCUS Challenge (MICCAI 2026): single-frame visual question answering about foreign objects in minimally invasive surgery.

Official pre-evaluation score: 59.11.

What is distinctive about it

The adapter targets 16 module names, matching 614 modules, including two families that a conventional LoRA target list misses entirely on this architecture:

• the vision encoder (qkv, proj, linear_fc1, linear_fc2);
• the Gated DeltaNet linear-attention layers (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj) — with full_attention_interval = 4, these are 48 of the 64 token mixers, 21 % of the model's parameters.

Under the conventional list both receive zero gradient. Adding the vision-encoder names alone was worth +3.87 points on the official metric in a matched ablation — more than a 3.4× increase in base-model size.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3.6-27B-FP8"      # or Qwen/Qwen3.6-27B for bf16
ADAPTER = "Potestates/orena-focus-frame-qwen3.6-27b-lora"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda:0")
model = PeftModel.from_pretrained(model, ADAPTER)   # do NOT merge_and_unload()
model.eval()

image = Image.open("frame.png")                     # native resolution, no resizing
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text",  "text": "How many Clips are visible in this frame?"},
]}]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False)                          # required
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Three things that will otherwise cost you hours

1. enable_thinking=False is mandatory. With the default template the model emits an extended reasoning trace and never reaches an answer.
2. Do not merge the adapter into an FP8 base. It targets quantised modules; use it unmerged. The vision encoder is excluded from quantisation in the official FP8 checkpoint, so the vision-side LoRA applies losslessly either way.
3. Running the FP8 base needs a transformers fix. quantizers_utils.should_convert_module uses re.match, which anchors only the start of a module name. The checkpoint's modules_to_not_convert contains entries ending in mlp.gate, which prefix-match mlp.gate_proj, leaving all 64 gate_proj layers unquantised and their FP8 bytes read as bf16 with scales discarded. The symptom is fluent-looking garbage; the signature is that the only UNEXPECTED keys at load are gate_proj.weight_scale_inv. Use re.fullmatch. The bf16 base is unaffected.

Training data

• FOCUS FRAME training split — 13 748 QA rows over 92 videos (challenge-provided).
• Potestates/sar-rarp50-focus-qa — 1 616 QA rows derived from SAR-RARP50 human segmentation masks, used for 100 steps of domain correction.

No other data. No pseudo-labels, no synthetic data, no private data.

Results

Limitations

Counting (aggregation) is the weakest capability. The model compresses the numeric range — regression slope of prediction on truth ≈ 0.51–0.59, and it never emits a value above 8. Probing indicates the true count is not recoverable from the representation, so post-hoc recalibration is capped at about +0.9 points. See the method description for the full analysis and twelve interventions that did not work.

The stage-2 domain correction is effective but blunt: it works partly by collapsing toward the majority class on out-of-distribution anatomy, and costs 2.62 points on in-distribution object recognition.

Links

• Method description and code: https://github.com/DingTianxingjian/orena-focus-frame
• Derived annotations: https://huggingface.co/datasets/Potestates/sar-rarp50-focus-qa
• Challenge: https://frame.orena-focus-challenge.org/

Licence

Released under CC BY-NC-SA 4.0 for non-commercial research use. The base model is Apache-2.0; the non-commercial restriction is inherited from the challenge training data, part of which is distributed under CC BY-NC-SA.