nara-qwen-3.5-2b

nara-qwen-3.5-2b

Qwen3.5-2B fine-tuned for cultural heritage transcription of English-language archival documents — United States National Archives Revolutionary War pension files: printed forms completed by hand, catalogue cards, and pages of 18th- and 19th-century American cursive.

Given a page image the model returns one line of text per physical written or printed line, reading the printed text and the handwriting together in the order they appear.

This is the finished 40,000-row model — all 3 epochs, 14,535 steps.

Provenance — a medieval Latin model retrained on English archives

1. Qwen/Qwen3.5-2B — the stock vision-language model.
2. comma-qwen-3.5-2b-full — 9,906 pages of medieval Latin under the CATMuS graphemic standard (cer_nfd 0.1251).
3. NARA, 10,000 rows — 8,756 pages of this dataset, 3 epochs.
4. NARA, 40,000 rows — 38,756 pages, 3 epochs. This model.

The medieval ancestry is not incidental. Before NARA training the model applied CATMuS conventions to English: asked to read a pension form it wrote SERUICE for SERVICE, because rule B5 of its previous prompt maps v to u. Undoing that is a large part of what this fine-tuning did — and it is why this model ships its own prompt.txt rather than the CATMuS one.

Results

The full held-out split: 762 pages from 39 documents that appear nowhere in training (0 documents shared with the 1,407 training documents). A document is held out whole, because the corpus averages ~27 pages per document in the same hand on the same printed form, so splitting by page would measure memorisation rather than reading.

Against its parent: micro CER improved by 0.1124; macro CER is unchanged (0.4492 to 0.4607, inside the 0.02 noise floor); line recall is unchanged (0.1793 to 0.1911, inside the 0.02 noise floor).

Both columns were produced in the same job, on the same pages, by the same code, so they are an exact comparison rather than numbers from different runs. Greedy decoding, 3072 max new tokens, 2048 visual tokens per page, batch 4.

cer_nfd equals cer here to every digit: NFD normalisation is a no-op on English ASCII. It matters for this project's medieval Latin models and not at all for this one.

Read this as one collection, not as general English HTR. Every page — training and holdout alike — comes from US Revolutionary War pension files. The holdout is unseen documents, which is a real test of generalisation across hands and forms, but it is not a test of generalisation across archives.

The failure mode worth knowing about

Error on this material is not evenly spread, and the gap between the micro and macro figures above is the measurement of that. A typical page of connected prose scores far better than the headline; the error concentrates on sparse catalogue cards, where a model that has not learned to stop transcribes the printed grid rules until it hits the token ceiling. One such page — 43 reference characters against 6,141 generated — can contribute as much edit distance as a hundred ordinary pages, because micro CER is a ratio of totals.

Micro CER weights a page by its length, so it describes a long page of prose. Macro CER weights every page equally, so it describes the sparse cards. Quote the one that matches your material, and expect short, sparse pages to be considerably worse than the headline.

Two things reduce it: more training data (1,000 → 10,000 pages cut truncation from 15% to 2% on the 2B, and 10,000 → 40,000 cut it further to 0.9%), and repetition_penalty=1.1 at inference. Use the penalty when serving.

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "wjbmattingly/nara-qwen-3.5-2b"
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype=torch.bfloat16,
                                                    device_map="auto")
processor = AutoProcessor.from_pretrained(REPO)

# The prompt is part of the model. It ships in this repo as prompt.txt and must
# be used verbatim: the model was trained to answer this exact instruction, and
# a paraphrase is a different task than the one it was fine-tuned on. In
# particular, do NOT use the CATMuS prompt from this project's medieval models.
from huggingface_hub import hf_hub_download
prompt = open(hf_hub_download(REPO, "prompt.txt")).read()

image = Image.open("page.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True,
                                     enable_thinking=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=3072, do_sample=False,
                     repetition_penalty=1.1)
print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                                 skip_special_tokens=True))

enable_thinking=False is required, not optional. Qwen3.5's chat template leaves the thinking block open by default at 4B and above; a model trained on a closed block will then reason instead of transcribing and never reach the text.

The repo's processor_config.json is already capped to 2,048 visual tokens, the budget training and scoring used. Loading a stock processor instead shows the page at the library default of ~16k tokens — a resolution the model never saw.

Training

Full fine-tune (2,213,241,664 parameters, all of them), 3 epochs, 14,535 steps, learning rate 5e-5, effective batch 8, bf16, on one RTX PRO 6000 Blackwell (39 h).

Training data

wjbmattingly/si-test — 38,756 pages from 1,407 documents, the first 40,000 rows of the dataset. Targets are line-by-line transcriptions; --- marks a horizontal rule.

Intended use and limits

Built for cultural heritage and digital humanities work: making handwritten archival material searchable and readable at scale. It is a research artifact, not a production OCR service.

It was trained on one collection — US federal pension files in English — and should be expected to degrade on other hands, other languages, other document genres, and other centuries. It reproduces what is written, including the spellings, abbreviations and historical language of the source; the transcription of a historical record is not an endorsement of its contents. Do not treat its output as an authoritative reading of a legal or genealogical record without human verification — at this error rate a name or a date can be wrong.