comma-qwen-3.5-2b-full

comma-qwen-3.5-2b-full

Qwen/Qwen3.5-2B fine-tuned to produce CATMuS-compliant, line-by-line transcriptions of medieval Latin manuscript pages.

Given a page image, the model returns one line of text for each physical written line, in reading order. CATMuS is a graphemic standard: the sequence of letters and signs as written, reduced to the modern Latin alphabet, with no editorial intervention. The output is not a translation and not a normalised reading.

This is a full fine-tune — a standalone checkpoint, no adapter and no peft dependency at inference.

• Training data — 9,905 pages from comma-project/deep-jsonl
• Evaluation — 243 held-out pages, each a different hand, none seen in training
• Demo — comma-qwen-3.5-demo

Results

Greedy decoding, 3072 max new tokens, 2048 visual tokens per page, batch 4. "Base" is the stock Qwen/Qwen3.5-2B scored on the same pages in the same job — a base number from another job on another day is not a control.

CER/WER are micro-averaged (total edits ÷ total reference characters), so a long page outweighs a short one; the macro mean is given alongside because a gap between them says the errors are concentrated. line_recall is the share of reference lines reproduced exactly and in order — for a line-by-line CATMuS target that is the number a palaeographer looks at first.

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "wjbmattingly/comma-qwen-3.5-2b-full"
BASE = "Qwen/Qwen3.5-2B"

processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map="cuda"
)
model.eval()

# The prompt is the 11.6k-character CATMuS rule set the model was trained under.
# Serving a different prompt serves a different task: the rules are what the
# target obeys, so the model was taught to read them as part of the input.
prompt = open("prompt.txt", encoding="utf-8").read()

image = Image.open("page.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [{"type": "image"}, {"type": "text", "text": prompt}],
}]

# `enable_thinking=False` is not optional. With no kwarg, Qwen3.5 0.8B/2B leave
# the thinking block CLOSED and 4B/9B leave it OPEN -- and a model trained on
# pure transcription that is handed an open block will think instead of
# transcribing, emitting prose until the token budget runs out.
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
assert text.count("<think>") == text.count("</think>"), "thinking block left open"

inputs = processor(text=[text], images=[[image]], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        do_sample=False,             # greedy is the published control
        repetition_penalty=1.1,      # the measured serving default; 1.0 reproduces the table
        max_new_tokens=3072,
        eos_token_id=[processor.tokenizer.eos_token_id],
        pad_token_id=processor.tokenizer.pad_token_id,
    )

print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                                 skip_special_tokens=True).strip())

prompt.txt is in the demo Space.

The image budget travels with the checkpoint

This repo's processor_config.json is already capped to 2048 visual tokens per page (2,097,152 pixels after smart-resize, one token per 32×32 block), which is what training and scoring used. Load the processor from this repo rather than from the base model: the library default is 16,777,216 pixels — 16k visual tokens for one page — and serving at that budget shows the model a page at a resolution it never saw.

Training

max_length never truncates: every sample is measured first and over-budget ones are dropped, because a truncated completion teaches the model to stop mid-transcription and a truncated prompt removes the rules the target obeys.

Limitations

• Nothing under 0.02 CER is a result. Four seeds of one configuration in this project gave CER 0.1271 / 0.1334 / 0.1370 / 0.1485 — mean 0.1365, sd 0.0090. bf16 training is nondeterministic across nodes and greedy decoding amplifies a sub-millivolt logit difference into a different token. Treat differences smaller than ~0.02 as ties.
• Greedy is deterministic given identical batching, but not batch-size invariant. Re-scoring the same pages at batch 1 instead of 4 changes almost every prediction and moves aggregate CER by ±0.013 for the smallest model here. Any comparison across a batch-size change is invalid.
• Quote cer_nfd, not cer. ũ (U+0169) and u + combining tilde are one glyph, and 77% of training targets mix the two forms — so an edit distance over raw code points charges two edits for a difference that is not on the page, about 16% of all edit operations. Every checkpoint is biased toward one form and training does not fix it (the targets are mixed, so there is no signal to fit), which means raw cer partly scores which Unicode form a tokenizer prefers. NFD and not NFC: t̃ r̃ m̃ p̃ c̃ q̃ have no precomposed form.
• The remaining error is mostly convention, not reading. On the best run of this project, word-division spaces are ~17% of all edit operations and allographs another ~5%, against ~1% for genuine letter confusion (r↔s, f↔s). Both are decisions about the transcription convention. Do not read a CER at this level as a statement about how well the model reads the script.
• Latin, and this convention. Trained on Latin-dominant manuscripts under the CATMuS graphemic standard. deep-jsonl uses the full MUFI superscript-letter repertoire (qͥ qui, qͣ qua, qͦ quo) where coarser conventions flatten almost everything to a tilde, so scoring this model against a corpus transcribed more coarsely costs 1–2% CER for reasons unrelated to reading.
• Blank and non-text pages return the literal string VLM-NO-TEXT.

Citation

@misc{comma_qwen35,
  title  = {comma-qwen-3.5: CATMuS transcription models for medieval Latin manuscripts},
  author = {Mattingly, William J. B.},
  year   = {2026},
  url    = {https://huggingface.co/wjbmattingly/comma-qwen-3.5-2b-full}
}