hanji-parse-4b

Hanji Parse 4B

Hanji Parse 4B is a document-parsing vision-language model fine-tuned from Qwen/Qwen3-VL-4B-Instruct. Given a page image, it emits a JSON array of layout-grounded content blocks — each block a semantic section of the page (a paragraph, a heading with its content, a key-value panel, or a whole table) with a bounding box and its transcribed text. Tables are transcribed as GitHub-Flavored Markdown inside a single block. Figures, photos, and signatures are returned as image blocks.

Output contract

The model returns JSON only: an array of records

[{"bbox_2d": [x1, y1, x2, y2], "text_content": "..."}]

• bbox_2d is [left, top, right, bottom] in normalized 0–1000 page coordinates (divide by 1000 and multiply by the page width/height to recover pixel boxes).
• Blocks are semantic sections, typically 5–30 per page — not one record per line, cell, or field.
• Tables come back as one block containing a GitHub-Flavored Markdown table; every cell rides inside the markdown.
• Non-text graphics (photos, charts, stamps, signatures) come back with text_content = "<image>".
• Checkboxes are transcribed inline as [x] / [ ] before their label.
• An empty page returns exactly [].

Usage — read this before running the model

1. Image preprocessing

• Downscale so the image is at most 2,000,000 pixels (2 MP), preserving aspect ratio. Never upscale.
• Floor each dimension to a multiple of 32 (Qwen3-VL uses 16-px patches with 2×2 spatial merge → 32 px per visual token).
• Use LANCZOS resampling; feed the result as a PNG.

from PIL import Image

MAX_PIXELS, PATCH = 2_000_000, 32

def preprocess(img: Image.Image) -> Image.Image:
    w, h = img.size
    scale = min(1.0, (MAX_PIXELS / (w * h)) ** 0.5)
    w, h = int(w * scale) // PATCH * PATCH, int(h * scale) // PATCH * PATCH
    return img.convert("RGB").resize((w, h), Image.LANCZOS)

2. The prompt

Send the page image followed by exactly this text as the user turn. Do not paraphrase, extend, or reformat it.

Detect every BLOCK in this document and return a JSON array.

Schema: [{"bbox_2d":[x1,y1,x2,y2], "text_content":"..."}]
Coordinates: normalized 0-1000 page coordinates; [x1,y1,x2,y2] = [left,top,right,bottom].

EMPTY PAGE: If the page has no legible content, return exactly []. Otherwise, transcribe every legible content block; a page with only one legible item is not empty.

Your DEFAULT is to GROUP. Most such pages form 5-30 records; sparse pages may form only 1-4. More than 30 remains unusual. A
"block" is a semantic SECTION (a panel, a heading + its content, a key-value group,
or a whole table), NOT a single line, cell, or field. If you are emitting one record
per line, per cell, or per form field, STOP - that is WRONG. When a region is not a
clean table, you must STILL group it into section blocks; never fall back to
one-record-per-element.

Block categories:
- Text records: ONE record per block - a heading TOGETHER WITH the lines beneath it,
  a paragraph, a list, or a key-value field group. text_content = the block's text,
  with "\n" between its lines. DO NOT emit one record per line.
- Table records: a table or dense grid of cells, rendered as GitHub-Flavored Markdown
  (| col | col |\n|---|---|\n| cell | cell |). Never emit one record per cell or
  per row - the markdown carries every cell. A repeated item|amount list (receipt
  lines, menu items) IS a table.
  HEADERLESS TABLES: if a table has no visible column headings, DO NOT invent any.
  Render only the visible rows/cells in their observed order. If Markdown syntax needs
  a separator row, use empty header cells rather than synthetic names like "Column 1".
  TABLE CELL TEXT: cell contents must be plain visible text. Do NOT add Markdown
  emphasis or formatting inside cells (no **bold**, _italics_, backticks, or headings)
  unless that formatting is the only way to preserve information that is visible on
  the page.
  TALL TABLES: A logical table on one page is ONE block regardless of row count. Include
  every visible row in one GFM table. Its bbox_2d must tightly enclose the full table
  from the first row through the last row.
- Image records: ONE record per photo, figure, chart, scan, or non-text graphic.
  text_content = "<image>". Handwritten signatures, cursive e-signatures, initials,
  signature scribbles, and signature marks are ALWAYS images - do NOT transcribe or
  guess them, even if partly readable. Printed labels such as "Signature:" remain text.
  Do NOT emit for logos < 40 px wide.

CRITICAL - transcribe MEANING, not layout glyphs:
- Fill-in / blank lines: emit ONLY the label, NOT the blank. Write "Name:" - never
  "Name:________________". For signature fields, keep the printed label as text and
  emit the actual signature mark itself as an image record with text_content="<image>".
- NEVER reproduce decorative rules or separators - rows of *, -, _, =, ., or any
  repeated glyph. Omit them entirely; they are not content.
- Checkboxes / Y-N / selection fields: write the field and its options on one line with the marks inline - "<row label> Y [x] N [ ]", "<label>: [x] Yes [ ] No"; "[x]" filled, "[ ]" empty; one record per field, not per option.
- text_content must equal the VISIBLE text of the block - never pad, repeat, or
  continue a character run. No single text block exceeds ~20 lines; split a longer
  section at its sub-headings.

Grouping rules:
- A section heading and the content beneath it (down to the next heading) form ONE block.
- A field label and its value are ONE block - EVEN in a dense report header. Write
  "Visit Date: 08/12/2025" as one record; NEVER split the label from its value
  ("Visit Date:" + "08/12/2025" as two records is WRONG).
- On forms, when a key has an associated value, merge the key and value into the same
  block and bbox so the association is explicit. If several related key-value fields
  are visually grouped, emit the group as one block with one "Label: value" line per field.
- A bordered or visually-grouped PANEL (e.g. a "PRESCRIBER INFORMATION" box with all
  its fields) is ONE block - join its label:value pairs with "\n".
- A row of related cells that is NOT a clean table (a lab-result line, a transaction
  line) is ONE block - join the cells into one line; NEVER one record per cell.
- NEVER emit a bare value, a single cell, or a lone field as its own record.
- Keep COLUMNS separate: two side-by-side panels (e.g. Patient | Ordering Provider,
  Bill To | Ship To) are TWO blocks - never merge across the gutter.
- DO NOT merge unrelated neighboring panels or sections.
- Group by semantic relationship, NOT by bbox size. Small unrelated regions must stay
  separate. Example: a page title at the top-left and a page number at the top-right
  are TWO blocks, even if both boxes are small and on the same horizontal band. Only
  group items that belong to the same section, panel, list, table, or key-value group.
- Prefer CORRECT grouping over tight boxes: a block's bbox may be wide and may lightly
  touch a neighbor - do NOT over-split a section just to keep boxes small or separate.
- Page-edge content: if visible text touches or sits near the page boundary, inspect the
  full edge carefully and make the bbox include the entire visible glyphs/block, even if
  the box must start at 0 or end at 1000. Do NOT shrink edge boxes inward.
- Bbox coverage is strict: every transcribed character in text_content MUST be inside
  that record's bbox_2d, with no clipped letters. This matters most for small regions,
  rotated/non-horizontal text, page-edge text, headers/footers, stamps, and fax strips.
  Use a tight box around the actual region, but never make it so tight that any visible
  character you transcribed falls outside the box.

Output JSON only.
Checkboxes: transcribe every checkbox as [x] if marked or [ ] if unmarked, placed before its label (e.g. "[x] Allergies reviewed"; Y/N pairs as "Y [x] N [ ]"). Include every checkbox, including checkbox grids and Y/N option pairs.

3. Decoding

• Greedy: temperature 0.0, top_p 1.0, no repetition penalty.
• max_new_tokens = 8192.
• Strongly recommended: JSON-schema-constrained decoding (xgrammar in SGLang, guided_json in vLLM) with this schema:

{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "bbox_2d": {
        "type": "array",
        "items": {"type": "integer"},
        "minItems": 4,
        "maxItems": 4
      },
      "text_content": {"type": "string"}
    },
    "required": ["bbox_2d", "text_content"]
  }
}

Quickstart (transformers)

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL = "hanji-dev/hanji-parse-4b"
model = AutoModelForImageTextToText.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(MODEL)

image = preprocess(Image.open("page.png"))  # see preprocessing above
PROMPT = "..."  # the exact prompt above

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": PROMPT},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Serving (SGLang)

N-gram speculative decoding gives a large decode speedup on this output format (the repeated JSON keys draft extremely well):

python -m sglang.launch_server \
  --model-path hanji-dev/hanji-parse-4b \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 8192 \
  --speculative-algorithm NGRAM \
  --speculative-num-draft-tokens 16 \
  --speculative-ngram-max-bfs-breadth 10 \
  --enable-deterministic-inference \
  --context-length 16384

A ready-to-run server that implements the full preprocessing + prompt contract (and a schema-extraction API around it) is available at https://github.com/youlearn-ai/hanji.

License

Apache-2.0. Fine-tuned from Qwen/Qwen3-VL-4B-Instruct (Apache-2.0, © Alibaba Cloud / the Qwen team).