herculaneum-legibility-proxy

Herculaneum legibility proxy (proxy_v4 / proxy_s2s3_v1)

A small ResNet-18 binary classifier that scores ~1 cm windows of Vesuvius Challenge ink-probability maps for legible-text likelihood: does this window contain connected Greek letterforms, or just fiber texture / noise / damage?

It is a triage tool, not an ink detector. It runs downstream of the official ink-detection models, on their output maps, and answers a different question: "of all this detected ink, where should a human look first?"

Built and used to produce a full legibility index of Scroll 1 (PHerc. Paris 4): 125,298 windows scored across all 78 of Scroll 1's currently-published official ink maps (complete coverage, verified live against the S3 bucket 2026-07-08 — see "Coverage" below for how this grew from an initial 16-panel pass). 5,678 high-confidence text windows on the deduplicated wrap-series union (w010–w129); classic-lineage segments add 2,460 more on 18 physical surfaces (kept separate — their overlap with the series is unresolved), and 7,248 further gold windows come from redundant re-renders, counted only as consistency checks (see Coverage below). The same checkpoint, unmodified, was then run on three more scrolls with no fine-tuning in between:

Coverage — how the Scroll 1 index reached 78/78

The index originally covered only 16 of Scroll 1's 81 public segments (one continuous panel series, w010–w100, uploaded 2026-06-23) — 29,853 windows, 4,491 gold. A live S3 check on 2026-07-08 found the scroll had grown to 78 populated official maps (3 of the 81 public segments still have none). The remaining 62 were scored with the identical unmodified pipeline (same proxy_v4.pt, same WIN=512/STRIDE=256 grid, same ds8-resolution preprocessing — verified byte-for-byte SHA256-identical against the official pre-made ink-detection/downsampled/*-ds8.jpg product before trusting it on new segments), split into four groups:

Correction (2026-07-08, caught in adversarial review): the _jordi panels were originally folded into the "distinct" count because their segment IDs looked like part of the w101–w129 extension. They aren't — they cover wrap ranges already present in the original 16-panel index. Moved to the overlap bucket; the headline distinct-territory number dropped from an incorrect 8,970 to a correct 8,231 as a result. The July-2026 series is confirmed not a byte-identical duplicate of the June one (e.g. the w010-027 file is 682 MB in June vs 1.07 GB in July, same recipe tag), but it covers the same physical wraps, so — like the _jordi panels — its gold windows are reported separately rather than folded into the "distinct territory" total, to avoid double-counting the same underlying text. [Update, 2026-07-11: with one exception found later — the July w104-106 panel is unique territory; see the correction below.] Both overlap groups are useful anyway as independent consistency checks: wrap range w028-037 (June vs July) scores gold at 50.9%/40.0%, and w046-052/w053-058 (original vs _jordi) show matching gold-density fingerprints band-by-band — unrelated renders of the same regions agreeing on where the text is.

Correction (2026-07-11, peer review): an earlier version of this card reported "8,231 gold on distinct territory" by summing the wrap-series panels with the classic segments. Two problems, both fixed here: the June w104-106 segment has no published ink map, so the July panel covering those wraps is unique territory (131 gold) wrongly excluded as redundant; and the classic bucket contained re-renders of the same physical surfaces (a _copy, a _v14, a _v2_flatboi, a _v8, an offset-0 re-render — 23 panels are 18 surfaces) and its physical overlap with the panel series is unresolved. Canonical accounting: deduplicated wrap-series union (June w010–w129 + July w104-106) = 5,678 gold windows; classic-lineage representatives = 2,460 gold on 18 surfaces, reported separately; redundant re-renders = 7,248 gold (224 classic + 739 _jordi + 6,285 July), used only as consistency checks. Cross-check invariant: 5,678 + 2,460 + 224 + 739 + 6,285 = 15,386 gold rows in the full 125,298-window index — shipped as full_index_complete.json in the companion dataset repo (SHA256 4d393d70ce886ed62b7e73e365f1d01cbe7f6efa37168fb3f27ade2b89d6e7a8; every bucket and denominator above is recomputed with asserts by its summarize_index.py). Note the score key in that file is named v3 for legacy reasons; every score in it was produced by proxy_v4.pt.

Files

Both are plain state_dicts for torchvision.models.resnet18() with fc = nn.Linear(512, 1) (122 tensors).

Which one to use — the split is by rendering pipeline, not by scroll

proxy_v4 generalizes across scrolls, as long as the map came out of the official/community rendering pipeline (the texture statistics of an "official-style" ink map are consistent regardless of which scroll it's of). See the table above for the four-scroll validation: one native, one blind positive control (0139), one cross-render-calibration caveat that's about the ink detector, not this classifier (Scroll 5), and one check against fully independent, non-visual ground truth — transcribed-letter counts, not just human eyeballing (Scroll 4; project-internal validation note — the correlation, n, and p reported above are the complete result).

proxy_s2s3_v1 exists because that assumption breaks on maps rendered by a custom / non-official pipeline — which for us happened to be Scrolls 2 and 3, only because no official ink maps existed yet for the regions we needed and we rendered our own. On those self-rendered mosaics, proxy_v4 alone gives an inflated background-noise floor (~0.04); the 192 self-rendered-domain negatives used to fine-tune proxy_s2s3_v1 drop that to ~0.003 without hurting Scroll 1 performance (AUROC 0.984 vs 0.985). It was never evaluated on official maps of other scrolls (0139, Scroll 5) — there's no reason to prefer it over proxy_v4 there, since proxy_v4 already has evidence of working on that domain and proxy_s2s3_v1 doesn't.

How to use

The preprocessing is part of the model. Deviate from it and the scores shift silently — this exact recipe is what the training used:

import numpy as np
import torch, torch.nn as nn, torchvision
from PIL import Image

m = torchvision.models.resnet18()
m.fc = nn.Linear(512, 1)
m.load_state_dict(torch.load("proxy_v4.pt", map_location="cpu"))
m.eval()

def score_window(win_u8):          # win_u8: 2D uint8 crop of an ink map
    c = win_u8.astype(np.float32)
    active = c[c > 10]
    if len(active) > 50:           # percentile stretch over active pixels
        lo, hi = np.percentile(active, [2, 99.5])
        c = np.clip((c - lo) / max(hi - lo, 1e-6), 0, 1)
    else:
        c = c / 255.0
    img = Image.fromarray((c * 255).astype(np.uint8)).resize((160, 160), Image.BILINEAR)
    t = torch.from_numpy(np.array(img, np.float32) / 255.0)[None].repeat(3, 1, 1)
    with torch.no_grad():
        return torch.sigmoid(m(t[None])).item()

The window must be ~1 cm physical — this is the #1 usage error

The model was trained on 512 px windows of ds8 panels at ≈19.2 µm/px ≈ 9,830 µm ≈ 1 cm per window. On any other map, compute the window size from the segment's real physical scale, e.g.:

px_um = (area_cm2 * 1e8 / (H * W)) ** 0.5   # from the mesh meta.json area
WIN   = round(9830 / px_um)

Two failure modes we hit ourselves, so you don't have to:

1. Do not assume 512 px. On a 7.91 µm-scan ds8 map the right window is ~155 px; on a 2.4 µm-scan ds8 map it's ~512–560 px.
2. Do not stretch undersized crops. If a fragment is smaller than WIN, resizing what you have up to 160×160 changes the fiber texture's apparent frequency and inflates scores (we measured score anti-correlating with true window size, r ≈ −0.45, before fixing this). Pad onto a black 160×160 canvas at the training scale (61.4 µm per classifier pixel) instead, and treat such windows with extra suspicion.

Thresholds we used

• ≥ 0.9 — "gold": on Scroll 1, round-2 human review confirmed 120/120 of the model's top-ranked proposals. On PHerc 0139, all 63 gold windows were human-reviewed (single reviewer): 54 clear text (85.7%), 9 possible, 0 rejected as noise/artifact. These are one reviewer's visual verdicts on model-selected windows, not population precision estimates.
• 0.35–0.70 — genuinely ambiguous band; useful for uncertainty sampling if you're fine-tuning.
• Scores are not calibrated probabilities, and they do not transfer across ink-model renders: on Scroll 5, the same three candidate windows (segment w066 and neighbors) scored 0.938 / 0.914 / 0.903 on july_retreat-model maps but 0.266 / 0.029 / 0.334 on november19-model maps of the identical windows — pixel correlation between the two maps in those windows is 0.86–0.89 (i.e. both detectors render essentially the same visible letterforms; only the score disagrees). Rank within one map family; never compare raw scores across families, and never take an AND-style agreement filter (min(scoreA, scoreB) ≥ 0.9) at face value without checking whether one model is just systematically deflated — we lost 3 true positives to exactly that before catching it.

Training summary

• Architecture: torchvision ResNet-18 (ImageNet init), fc → 1, BCE-with-logits, AdamW.
• Data (proxy_v4): 541 positive windows from three rounds of human-in-the-loop review of official Scroll 1 panel maps (393 browse-all
  • 120 model-proposed/human-confirmed + 28 uncertainty-sampled), negatives from 50 GPU-verified fiber windows (shipped in the companion dataset repo as fiber_negatives_50.jsonl; 12 of the 50 fall in the validation panels), background sampling, and 87 human-implied negatives at weight 0.7; 22 human-"unsure" windows excluded. Spatial validation: 3 full held-out panels → AUROC 0.985. (Corrected 2026-07-08: the browse-all round produced 398 raw rows, but 5 of those were tagged "unsure," not positive — 393 + 120 + 28 = 541. The full, independently re-verified breakdown ships as training data, not just this prose summary — see the companion dataset repo: https://huggingface.co/datasets/LimeGS/herculaneum-legibility-proxy-labels.)
• proxy_s2s3_v1: initialized from proxy_v4, fine-tuned at LR 1e-4 with self-rendered Scroll 2/3 negatives: 23 human-reviewed noise windows at weight 1.0, plus ~169 additional implicit negatives at weight 0.7 from the same rendering pass. The 7 human-flagged candidate windows were excluded from training entirely. Reproducibility caveat: only the 23 explicitly-reviewed windows are traceable in this release — the ~169 implicit negatives were read at training time from a session-scratchpad file that no longer exists, so they can't currently be regenerated bit-for-bit. This doesn't change what was trained (the weights are final), it means the companion dataset repo can reproduce 23 of the ~192 S2/S3 negatives, not all of them — noted there in full, not glossed over here.
• Trains in minutes on a laptop (Apple MPS, CUDA, or CPU); this is deliberately a small, reproducible model.

Training data release

The human-labeled training coordinates (not images — just {panel, y, x, label, weight}, since the underlying maps are already public) are released alongside this model at https://huggingface.co/datasets/LimeGS/herculaneum-legibility-proxy-labels, with a crop-regeneration script, a consolidated training script, pinned dependency versions, and a self-test. Scope of reproducibility, stated precisely: the full proxy_v4 recipe (labels + 50 fiber negatives + procedural background negatives → crops → training with the original spatial holdout) can be re-run end to end from public data; expect a checkpoint that matches in architecture and closely in behavior, not bit-for-bit (GPU/RNG/environment nondeterminism; a retrained checkpoint's AUROC will vary around 0.985, while the PUBLISHED checkpoint's validation AUROC is exactly reproducible: the dataset repo's eval_checkpoint.py deterministically rebuilds the held-out split, 81 positives / 123 negatives, and yields 0.9853457794). proxy_s2s3_v1 additionally has the documented 169-negative gap. The scoring INDEX built with the model ships in the same dataset repo (full_index_complete.json + summarize_index.py); the sweep script that produced it is project-internal, but its recipe is fully specified above (the score_window() preprocessing on a WIN=512 / stride-256 grid).

Limitations / what this does not claim

• Not an ink detector. Input is an ink-probability map produced by the official/community detectors, never raw CT.
• Finds letterform-like structure, not meaning. A high score means "connected strokes at letter scale," which is what a human reviewer should look at — it is not a transcription and not proof of novel text.
• Domain is defined by rendering pipeline, not by scroll identity — see "Which one to use" above. proxy_v4 is validated on official-style maps from four scrolls (S1, PHerc 0139, Scroll 5, Scroll 4); a fifth scroll's official maps haven't been tried, though the pattern across four independent scrolls (including one quantitative correlation against transcribed-text ground truth, not just spot checks) makes transfer the expected outcome, not a hopeful one.
• Cross-render score shifts are real even within proxy_v4's validated domain: two different official ink-detection models of the same scroll region can disagree by 0.1+ (see Thresholds) — that's the ink detector's output changing, not this classifier's domain breaking.
• Windows flagged "gold" on public maps are locations worth expert attention, on data anyone can download — whether the text there is actually untranscribed must be checked against the papyrological literature, which is a moving target.

Provenance & license

Trained exclusively on publicly released Vesuvius Challenge data (CC BY-NC 4.0); the weights are released under the same CC BY-NC 4.0 to match. Human labels by the project's own reviewer. No official-team code or checkpoints are redistributed here.