Downloads · 30 days
11
18% of all-time downloads
octavigrau/museumscat-trocr-glyph-scorer
museumscat-trocr-glyph-scorer is a image-to-text model from octavigrau. Use it when you need a caption or text from an image. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
This is the final auxiliary handwriting checkpoint used in the fourth-place solution to the MuseumSCAT Specimen Collection Annotation Task.
Downloads · 30 days
11
18% of all-time downloads
All-time downloads
60
Public
Parameters
385M
1.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.5 GB · 100%
From the Hugging Face model README
This is the final auxiliary handwriting checkpoint used in the fourth-place solution to the MuseumSCAT Specimen Collection Annotation Task.
The model was not used to generate the final locality transcription freely. Its intended use is candidate-conditioned scoring: given a locality crop and several strings proposed by vision-language models, it estimates which candidate is most strongly supported by the visible glyphs. The complete solution code is available in octavigrau/kaggle-museumSCAT.
The checkpoint was initialized from Riksarkivet/trocr-base-handwritten-hist-swe-2, an Apache-2.0 TrOCR checkpoint for historical Swedish handwriting.
The final training mixture contained:
The test pseudo-labels were produced automatically from high-confidence model agreement; they were not manually transcribed. No MuseumSCAT images, crops, labels, or pseudo-label manifests are distributed in this model repository.
The MuseumSCAT competition data is provided under CC BY-NC 4.0. Users should also acknowledge the competition and the Natural History Museum of Denmark as required by the competition rules.
2e-5, weight decay 0.01;16;4.46.3.The checkpoint was selected using candidate-selection accuracy. Adding the pseudo-labeled crops moved the held-out result from 33/40 to 34/40 selections, a noise-sized improvement. In the competition pipeline, replacing the VLM medoid broadly with the TrOCR choice was harmful; the model was useful as a confidence signal and inside a strict correction gate.
For an image crop $x$ and candidate string $y$, the score is mean next-token log-likelihood:
$$ s(y\mid x)=\frac{1}{|y|}\sum_t \log p(y_t\mid y_{<t},x). $$
import torch
from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
checkpoint = "octavigrau/museumscat-trocr-glyph-scorer"
processor = TrOCRProcessor.from_pretrained(checkpoint)
model = VisionEncoderDecoderModel.from_pretrained(checkpoint).eval()
@torch.inference_mode()
def score_candidate(image: Image.Image, candidate: str) -> float:
pixels = processor(images=image.convert("RGB"), return_tensors="pt").pixel_values
target = processor.tokenizer(candidate, return_tensors="pt").input_ids
encoded = model.encoder(pixel_values=pixels).last_hidden_state
logits = model.decoder(input_ids=target, encoder_hidden_states=encoded).logits[:, :-1, :]
next_tokens = target[:, 1:]
token_scores = logits.log_softmax(-1).gather(-1, next_tokens.unsqueeze(-1)).squeeze(-1)
return float(token_scores.mean())
scores = {candidate: score_candidate(Image.open("locality_crop.png"), candidate)
for candidate in ["Ørslev", "Ørholm"]}
print(max(scores, key=scores.get), scores)
Higher scores are preferred. Scores are meaningful for comparing candidates for the same crop; they should not be interpreted as calibrated probabilities across images.