SwarmDo-A2

SwarmDo-A2 — one model, both skills (code and vision)

SwarmDo-A2 is an open, self-hostable, multimodal coding agent that fixes real software bugs and writes code from images — in a single model. It combines SwarmDo-A1's execution-verified coding delta with a teacher-free, render-verified visual-coding delta, folded together into one LoRA adapter on Qwen/Qwen3.6-27B. Built in Australia; measured, not marketing — we publish the negatives and the confidence intervals.

Public model hub & full results: https://github.com/SwarmDo/models

Keywords: open-source coding agent · agentic software engineering · SWE-bench-style patch generation · execution-verified best-of-N selection · image-to-code / chart-to-code / visual coding · render-verified self-improvement · multimodal vision LLM · Qwen3.6-27B LoRA / PEFT adapter · self-hostable on vLLM · Apache-2.0.

What A2 is (and how it differs from A1)

SwarmDo-A1 is the coding tier — a re-based open base plus a hardened agentic harness that verifies patches by running the project's tests. SwarmDo-A2 takes that same coding delta and combines it, on the same 27B base, with a separately trained visual-coding delta — a render-verified flywheel that turns a chart / plot / UI screenshot into the code that reproduces it and checks itself by rendering the output and comparing to the target image (an objective, hard-to-hack reward). The two deltas are fused into one rank-48 adapter, so A2 is a single checkpoint that carries both skills rather than two models you swap between.

The adapter reaches the architecture's gated-DeltaNet sequence-mixing layers (in_proj_*, out_proj), not just the standard attention projections — which is why A2 must be served merged (see Usage).

What's decision-grade (paired, held-out, execution-verified)

The headline claim for A2 is the churn gate: folding two independently trained skills into one model must not degrade either skill. We measured both, paired and held-out.

Inherited from the SwarmDo line (see A1): re-basing to Qwen3.6-27B ≈ +25 pp (p = 0.013); base + execution-verified selection solves 36 vs 24 (p = 0.0005); visual coding is image-driven (image-on 15 vs image-off 0, p = 6e-05).

These are relative, held-out, execution-verified comparisons on our own task sets — not an absolute public-leaderboard number we can't stand behind under paired testing.

Methodology note we're proud of: our harness can self-judge a patch (an LLM reading its own diff). We found that signal is ~2× optimistic, so every SWE number above is re-derived by actually executing the project's tests against the generated patch — never by self-grading.

Honest limitations (published, not buried)

• As a single model, A2 is ~neutral vs the base on raw coding — the same "churn wall" trade-off A1 documents. A2's value is (1) carrying both skills in one model with no measured degradation of either, and (2) the system around it (execution-verified best-of-N selection + render-verified visual coding).
• The SWE churn arm is deliberately small (40 paired tasks, 1 discordant pair): it rules out gross coding regression (39/40 verdict agreement, 40% byte-identical patches) but is underpowered for a subtle difference. The evidence for "coding preserved" is the near-identity, not the p-value.
• Python-centric agentic SWE + chart/plot/UI visual coding. Like all coding agents it can produce plausible-but-wrong patches — always run the project's tests against its output. Not a general chat assistant.

Usage — serve merged (important)

A2's adapter targets the base's gated-DeltaNet modules, which JIT-hang when served as a vLLM LoRA. Serve A2 as merged weights (native fast path). Merge with the VL-aware helper (it preserves the vision tower and emits the image-processor config so vLLM can serve it):

# 1) merge the A2 adapter into the base, keeping the vision tower
python merge_lora_vl.py --base Qwen/Qwen3.6-27B --lora ./ --out ./swarmdo-a2-merged

# 2) serve the merged model (multimodal + tool-calling)
vllm serve ./swarmdo-a2-merged --served-model-name swarmdo-a2 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --enforce-eager --trust-remote-code

merge_lora_vl.py is included in this repo. For one-step serving with no merge, use the pre-merged weights → SwarmDo/SwarmDo-A2-merged:

vllm serve SwarmDo/SwarmDo-A2-merged --served-model-name swarmdo-a2 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --enforce-eager --trust-remote-code

Or load with PEFT for local experimentation:

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "SwarmDo/SwarmDo-A2").merge_and_unload()
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.6-27B", trust_remote_code=True)

Serving notes: use --tool-call-parser qwen3_xml (not hermes — it silently drops tool calls) with --enable-auto-tool-choice; serve in eager mode if you hit hybrid-attention graph-capture issues; the linear-attention base keeps a fixed-size recurrent state, so long contexts cost prefill tokens, not growing KV cache. A text-only GGUF is not offered for A2 — quantizing to GGUF drops the vision tower, which would make the visual half inert. For code-only local runs, use SwarmDo-A1-GGUF.

Use cases

• One model for both automated bug-fixing and image-to-code — no adapter swapping.
• Automated bug fixing in Python repositories — generate a patch, verify it against the project's tests.
• Image-to-code / chart-to-code / screenshot-to-code — reproduce a chart, plot, or UI as working code, checked by rendering and comparing.
• Agentic software engineering — a coding agent that edits files, runs commands, and self-checks patches.
• Self-hosted / private multimodal coding assistant — runs entirely on your own hardware via vLLM; no data leaves your machine.
• Best-of-N patch generation — sample multiple candidate fixes and select the one that passes execution.

License

Apache-2.0, consistent with the base model family.

Citation

@software{swarmdo_a2_2026,
  title  = {SwarmDo-A2: One open model for execution-verified coding and render-verified visual coding},
  author = {SwarmDo},
  year   = {2026},
  url    = {https://github.com/SwarmDo/models}
}

FAQ

What is SwarmDo-A2? An open, self-hostable multimodal coding agent that both fixes software bugs and writes code from images, in a single model — the coding (A1) delta and a render-verified visual-coding delta fused into one LoRA on Qwen/Qwen3.6-27B.

How is A2 different from A1? A1 is the coding tier (adapter + execution-verified harness, with a text GGUF for easy local runs). A2 folds a separately trained visual-coding skill into the same model, so one checkpoint does code and vision — with measured proof that combining them degrades neither skill.

Is it open source? Yes — Apache-2.0, commercial use permitted. Weights, recipe, and full (including negative) results are public.

Can I run it on my own hardware? Yes — self-host with vLLM on the merged weights (see Usage: a one-line merge, then vllm serve). Full multimodal serving needs a GPU sized for a 27B; a driver new enough for the base's CUDA build is required.

Why must it be served merged, not as a LoRA? The adapter reaches the base's gated-DeltaNet layers, which JIT-hang under vLLM's LoRA path. Merged weights use the native fast path.

Is there a GGUF? Not for A2 — GGUF conversion drops the vision tower, making the visual half inert. Use A2 merged (vLLM) for code+vision, or the A1 GGUF for code-only local runs.

Does it really do image-to-code? Yes — it writes code from images and we verify it by rendering the output and comparing to the target image, an objective signal, not eyeballing. The visual skill is a teacher-free flywheel that measurably compounds as we scale its self-generated fuel.

Is it good at SWE-bench? We report relative, paired, held-out, execution-verified results (and re-derive every SWE number by running real tests, not self-grading), not a single absolute leaderboard number we can't stand behind.

What languages? Python-centric agentic software engineering, plus Python visual coding (charts/plots/UI). It is a coding agent, not a general chat assistant.

Where are the full results? Public hub: https://github.com/SwarmDo/models

---

Topics: open source multimodal coding model · one model for code and vision · self-hosted AI coding assistant · agentic SWE-bench model · image to code / chart to code / screenshot to code · render-verified visual coding · execution-verified code generation · Qwen3.6-27B LoRA · run a coding LLM locally · Apache-2.0 code model.