Qwen3-VL-D2R-Item-Reader-2B-Instruct

Qwen3-VL-D2R-Item-Reader-2B-Instruct

A LoRA fine-tune of Qwen3-VL-2B-Instruct that transcribes a single Diablo II: Resurrected (D2R) item tooltip from a cropped screenshot into one structured JSON object.

• Base model: unsloth/Qwen3-VL-2B-Instruct
• Method: LoRA (vision + language layers) via Unsloth + TRL
• Task: image-text-to-text → strict JSON (constrained decoding)
• Languages: English tooltips

Intended use

Transcribing a single, cropped D2R item tooltip (English) into structured fields. It is a narrow, in-domain transcription model, not a general-purpose VLM and not a chat model.

Out of scope: general OCR, other games, non-tooltip images, open-ended Q&A.

Input format

The model expects one tightly cropped tooltip, preprocessed to a fixed, orientation-aware canvas — 896×512 (landscape) for wide crops, 512×896 (portrait) otherwise — aspect-preserving fit, black padding. This matches the training distribution; other sizes/aspect ratios degrade accuracy. Decode greedily (temperature = 0) and constrain the output to the JSON schema below.

User turn: Extract the item from this tooltip.

System prompt (verbatim):

Transcribe the single Diablo II: Resurrected item tooltip in the image into one JSON object. Read it line by line, top to bottom. Output only the JSON.

A tooltip is a stack of lines, e.g.:

```
Viper Bite                       ← line 1: the item name (rare → a random made-up name)
Colossus Blade                   ← line 2: the base type (present on rare/set/unique/crafted/runeword items)
Two-Hand Damage: 70 to 198       ← base stats: damage / defense / chanceToBlock / durability / quantity
Durability: 14 of 14
Required Dexterity: 110          ← required* lines
Required Strength: 189
Required Level: 67
Sword Class - Slow Attack Speed  ← attackSpeed line (verbatim)
+180% Enhanced Damage            ← mods: every remaining magical stat line, in order
+25 to Maximum Damage
Adds 15-30 Fire Damage
8% Life Stolen Per Hit
+20 to Strength
+30 to Life
Ethereal (Cannot Be Repaired)    ← sets ethereal = true
Socketed (2)                     ← sets sockets = 2
```

Read EVERY digit exactly — `+202%` is not `+2%`, `550` is not `55`. Never guess, complete, round, reorder, or fix spelling.

`baseType` — where the item's base (e.g. Long Sword, Monarch, Ring) comes from depends on the rarity:
- normal, superior, low (gray) and magic items have NO separate base line; the base is inside line 1 (the name). Strip any leading quality word (Superior / Cracked / Damaged / Low Quality / Crude); for magic the base sits between prefix and suffix: `<Prefix> <BaseType> <Suffix>`, e.g. "Cruel Long Sword of Maiming" → "Long Sword".
- all other rarities (rare, set, unique, crafted, and runewords) give the item a unique made-up name on line 1; the base is the separate line 2 — that goes in `baseType`. For a runeword, also put the rune sequence shown in apostrophes (e.g. `'TalThul'`) into `runes`.

`quality` and `rarity` are two different things:
- `quality` = base tier, read from the NAME TEXT only: crude (name says Cracked/Damaged/Low Quality/Crude), superior (name starts with "Superior"), else normal. Ignore the color here.
- `rarity` = the NAME COLOR: low (gray), normal (white), magic (blue), rare (yellow), set (green), unique (gold/tan), crafted (orange). Runewords are unique.

`beltSize` = the integer N in a "Belt Size: +N Slots" line (belts only) — its own field, NOT a mod; null otherwise.

`mods` = every REMAINING stat line, verbatim and in order (keep `+`, `-`, `%`, ranges like `15-45`, parentheses). Do NOT include anything already captured in another field (name, baseType, damage, defense, chanceToBlock, beltSize, durability, quantity, required* lines, attackSpeed, the ethereal/socketed line, the class-restriction line) or UI hints ("Ctrl + Left Click ...", "Right Click to Use", "Keep in Inventory to Gain Bonus").

Output schema

The model is trained for constrained / structured decoding against the following JSON schema (response_format → json_schema, strict: true). Field order mirrors the in-game top-to-bottom reading order.

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "baseType": {
      "type": [
        "string",
        "null"
      ]
    },
    "category": {
      "type": "string",
      "enum": [
        "weapon",
        "armor",
        "shield",
        "offhand",
        "jewel",
        "ring",
        "amulet",
        "charm",
        "gem",
        "rune",
        "material",
        "potion",
        "consumable"
      ]
    },
    "itemType": {
      "type": [
        "string",
        "null"
      ]
    },
    "quality": {
      "type": "string",
      "enum": [
        "crude",
        "normal",
        "superior"
      ]
    },
    "rarity": {
      "type": "string",
      "enum": [
        "low",
        "normal",
        "magic",
        "rare",
        "set",
        "unique",
        "crafted"
      ]
    },
    "runes": {
      "type": [
        "string",
        "null"
      ]
    },
    "damage": {
      "type": [
        "object",
        "null"
      ],
      "properties": {
        "oneHand": {
          "$ref": "#/$defs/range"
        },
        "twoHand": {
          "$ref": "#/$defs/range"
        },
        "throw": {
          "$ref": "#/$defs/range"
        },
        "smite": {
          "$ref": "#/$defs/range"
        }
      },
      "required": [
        "oneHand",
        "twoHand",
        "throw",
        "smite"
      ],
      "additionalProperties": false
    },
    "defense": {
      "type": [
        "integer",
        "null"
      ]
    },
    "chanceToBlock": {
      "type": [
        "integer",
        "null"
      ]
    },
    "beltSize": {
      "type": [
        "integer",
        "null"
      ]
    },
    "durability": {
      "type": [
        "object",
        "null"
      ],
      "properties": {
        "current": {
          "type": "integer"
        },
        "max": {
          "type": "integer"
        }
      },
      "required": [
        "current",
        "max"
      ],
      "additionalProperties": false
    },
    "quantity": {
      "type": [
        "object",
        "null"
      ],
      "properties": {
        "current": {
          "type": "integer"
        },
        "max": {
          "type": "integer"
        }
      },
      "required": [
        "current",
        "max"
      ],
      "additionalProperties": false
    },
    "classRestriction": {
      "type": [
        "string",
        "null"
      ]
    },
    "requiredDexterity": {
      "type": [
        "integer",
        "null"
      ]
    },
    "requiredStrength": {
      "type": [
        "integer",
        "null"
      ]
    },
    "requiredLevel": {
      "type": [
        "integer",
        "null"
      ]
    },
    "attackSpeed": {
      "type": [
        "string",
        "null"
      ]
    },
    "mods": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "ethereal": {
      "type": "boolean"
    },
    "sockets": {
      "type": "integer"
    }
  },
  "required": [
    "name",
    "baseType",
    "category",
    "itemType",
    "quality",
    "rarity",
    "runes",
    "damage",
    "defense",
    "chanceToBlock",
    "beltSize",
    "durability",
    "quantity",
    "classRestriction",
    "requiredDexterity",
    "requiredStrength",
    "requiredLevel",
    "attackSpeed",
    "mods",
    "ethereal",
    "sockets"
  ],
  "additionalProperties": false,
  "$defs": {
    "range": {
      "type": [
        "object",
        "null"
      ],
      "properties": {
        "min": {
          "type": "integer"
        },
        "max": {
          "type": "integer"
        }
      },
      "required": [
        "min",
        "max"
      ],
      "additionalProperties": false
    }
  }
}

Note on rarity: rarity is a function of the item-name color (gold vs. yellow vs. orange are close in hue). The model is trained on it and predicts it directly — treat the model output as the primary source. A pixel-based (HSV) color read of the name line is a useful fallback when the model is uncertain.

Example outputs

Unique ring

{
  "name": "Raven Frost",
  "baseType": "Ring",
  "category": "ring",
  "itemType": "Ring",
  "quality": "normal",
  "rarity": "unique",
  "runes": null,
  "damage": null,
  "defense": null,
  "chanceToBlock": null,
  "beltSize": null,
  "durability": null,
  "quantity": null,
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": null,
  "requiredLevel": 45,
  "attackSpeed": null,
  "mods": ["+221 to Attack Rating", "Adds 15-45 Cold Damage", "+18 to Dexterity", "+40 to Mana", "Cold Absorb 20%", "Cannot Be Frozen"],
  "ethereal": false,
  "sockets": 0
}

Runeword (base item + rune sequence)

{
  "name": "Insight",
  "baseType": "Scythe",
  "category": "weapon",
  "itemType": "Polearm",
  "quality": "normal",
  "rarity": "unique",
  "runes": "RalTirTalSol",
  "damage": {"oneHand": null, "twoHand": {"min": 71, "max": 138}, "throw": null, "smite": null},
  "defense": null,
  "chanceToBlock": null,
  "beltSize": null,
  "durability": {"current": 65, "max": 65},
  "quantity": null,
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": 87,
  "requiredLevel": 27,
  "attackSpeed": null,
  "mods": ["Level 15 Meditation Aura When Equipped", "+35% Faster Cast Rate", "+247% Enhanced Damage", "+9 to Minimum Damage", "213% Bonus to Attack Rating", "Adds 5-30 Fire Damage", "+75 Poison Damage Over 5 Seconds", "+5 to Critical Strike", "+5 to All Attributes", "+2 to Mana After Each Kill", "23% Better Chance of Getting Magic Items"],
  "ethereal": false,
  "sockets": 4
}

Magic charm (procedural name)

{
  "name": "Harpoonist's Grand Charm of Sustenance",
  "baseType": "Grand Charm",
  "category": "charm",
  "itemType": "Charm",
  "quality": "normal",
  "rarity": "magic",
  "runes": null,
  "damage": null,
  "defense": null,
  "chanceToBlock": null,
  "beltSize": null,
  "durability": null,
  "quantity": null,
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": null,
  "requiredLevel": 42,
  "attackSpeed": null,
  "mods": ["+1 to Javelin and Spear Skills (Amazon Only)", "+38 to Life"],
  "ethereal": false,
  "sockets": 0
}

Training data

• Synthetic tooltips (primary): procedurally rendered from D2R game-data tables (item bases, affixes, uniques, sets, runewords) and composited over real stash/inventory backgrounds. Rendering is calibrated to the in-game tooltip (font, colors, line spacing, panel, border, separator) to minimize the synthetic→real gap.
• Real captures: optional domain anchor (hand-verified in-game crops, oversampled). Currently disabled — this run is synthetic-only.

Each training example is: system prompt + user turn + the tooltip image on an orientation-aware canvas (896×512 landscape or 512×896 portrait), with the target JSON as the assistant turn.

Training procedure

• Framework: Unsloth + TRL SFTTrainer, LoRA on vision and language layers.
• Precision: bf16 with gradient checkpointing.
• Key hyperparameters: LoRA r=16 / alpha=16, learning rate 0.0001, linear schedule, effective batch via gradient accumulation, ~1 epoch(s) over the synthetic set.
• Hardware: AMD Radeon RX 9070 (gfx1201) under WSL2 + ROCm.

How to run

Use any OpenAI-compatible local server (e.g. llama.cpp / LM Studio) with the GGUF build (model + vision mmproj):

1. Crop the tooltip and fit it to the orientation-aware canvas (896×512 if wider than tall, else 512×896; aspect-preserving, black pad).
2. Send the image with the system prompt above and temperature = 0, constraining the response to the JSON schema.

Limitations

• Domain-locked: only English D2R tooltips, cropped and canvas-fit as above.
• Color/rarity: see the rarity note — the model predicts it, with a pixel-based (HSV) read as fallback.
• Tiny digits / very long tooltips can still be misread; verify critical values.
• No formal benchmark; evaluated qualitatively on held-out real captures.

License

Derived/LoRA weights are released under Apache-2.0, inheriting the base model's license (verify the exact terms of unsloth/Qwen3-VL-2B-Instruct before redistribution). Diablo II: Resurrected and all related assets/trademarks belong to Blizzard Entertainment; this is an unofficial, fan-made model with no affiliation or endorsement.

Acknowledgements

• Qwen3-VL-2B-Instruct team (base model).
• Unsloth for the fine-tuning stack.
• The blizzhackers d2data project for the game-data tables used to generate the synthetic training data.