semantic-image-arith-ckpts

semantic-image — arithmetic image-edit checkpoints

22 checkpoints from the experiments in semantic-image, selected because each one is a number in the results table — not because it is the best model. Several are here precisely because they fail.

The task: given a 512×512 image of Q: 21 + 60 = ? / A:, edit it so the answer appears after A:. Input and target differ only in the answer glyphs, so the pixel loss isolates arithmetic. The metric is Qwen3-VL-2B OCR accuracy on a 1,000-example held-out test set (the older nearest-neighbour render_acc proxy overstates it and is not used here).

What is here

Each folder is <family>/<run>/ and holds transformer/, meta.json, train_args.json (the exact argv) and CARD.md.

noemit-2x2 — does the DiT compute, or does the text encoder?

Pure MSE, no emitted token; the answer must be computed inside the network.

Zero train/test gap on the winner (88.0% train vs 87.7% test) — this is generalization, not memorization.

seeds — how much of that is seed noise

M4 across seeds 43/44/45: 35.8% / 89.9% / 88.8%. M3: 25.8% / 7.0% / 14.8%. Seed 43 for M4 is a genuine outlier, kept here so the variance is not hidden.

emit-budget — answer tokens as a compute budget

The text encoder emits K tokens that are re-encoded and fed to the DiT. K is a compute budget, and the response is monotone:

The pretrained DiT collapses at small K (it ignores a partially-hinted answer and falls back to rendering), and only catches up once the hint is nearly complete.

emit-slots — the same axis at 300 epochs

1 slot / answers ≤9, scratch: 99.2%. 2 slots / ≤99, pretrained: 89.2%. 3 slots / 0–198, pretrained: 69.7%.

t2i, holdout

The text-to-image port (no input image) reaches only 10.1% / 5.4% — the edit setting is doing real work. holdout is the train≠test split with the answer carried by the DiT + text decoder: 88.7%.

Missing on purpose

Five checkpoints that produced published numbers no longer exist: those runs continued past the evaluated step and --keep_last_ckpts pruned the older ones (including one explicitly named step_45000_ep10_KEEP). The affected numbers — emit3slots scratch 90.6% and 91.1%, emit3slots pretrained 60.0%, emit2slots scratch 92.1% — are reproducible from the launchers but not restorable as weights. Their eval reports and per-row OCR output survive in the dataset repo.

Loading

from diffusers import QwenImageEditPipeline
from diffusers.models import QwenImageTransformer2DModel

tr = QwenImageTransformer2DModel.from_pretrained(
        "SuhaoYu1020/semantic-image-arith-ckpts",
        subfolder="noemit-2x2/<run>/transformer", torch_dtype="bfloat16")
pipe = QwenImageEditPipeline.from_pretrained("Qwen/Qwen-Image-Edit", transformer=tr)

Weights only — no optimizer state. freeze_te runs ship the DiT alone (~3.8 GB scratch / ~38 GB pretrained); pair them with the stock Qwen-Image-Edit text encoder. Runs that trained the text encoder include it (~19 GB / ~53 GB).

diffusers must be the pinned dev build, git+https://github.com/huggingface/diffusers@6abf75263a09a3e7a62458f544ce2fac28568fe2.

Citation / code

Code, launchers and the full results table: the semantic-image repository. Datasets: SuhaoYu1020/semantic-image-arith.

Image-to-image arithmetic with different encoders (work in progress)

Two top-level directories, Qwen-Image-Arith-VanillaDiT/ and Qwen-Image-Arith-ScratchTE/, hold checkpoints of an ongoing study and follow a different layout from the rest of this repo: paths mirror projects/ of the code repository (<project>/runs/<run>/<checkpoint>/), with several checkpoints per run and the run's args.json next to them. All models generate the answer image with a from-scratch DiT trained with an image MSE loss on two-digit addition (arithmetic_add_unordered, or its font / size / stroke / position-augmented variant, …addunordAUG… in the run name). They differ in how the question image is read:

• Qwen-Image-Arith-VanillaDiT/runs/vanillaDiT_D28H2048_FRAME_…: no text encoder; the source image enters as VAE latents in a second frame.
• Qwen-Image-Arith-ScratchTE/runs/vitPretrainedTE_…: frozen Qwen2.5-VL vision tower, its 7B language model trained.
• Qwen-Image-Arith-ScratchTE/runs/vitScratchTE_…: frozen vision tower, a randomly initialised 12-layer language model (SCRATCHVITtrainvit: the vision tower is random and trained too).

Checkpoint names are training steps (4,500 steps per epoch at batch 2; 2,250 for the bs1ga4 runs). No accuracy is claimed here.

from huggingface_hub import snapshot_download
snapshot_download("SuhaoYu1020/semantic-image-arith-ckpts", local_dir="code/projects",
                  allow_patterns=["Qwen-Image-Arith-VanillaDiT/runs/*", "Qwen-Image-Arith-ScratchTE/runs/*"])