card-scanner-siglip-lora

Card scanner SigLIP2 — name-first release, 2026-10-12

A rank-16 LoRA adapter for the SigLIP2 vision tower, with a matching gallery of 57,913 products spanning English Pokémon, Japanese Pokémon, and One Piece. Used by card-scanner-siglip.

This release continues the combined-catalog adapter for 300 steps using exact_product_cross_entropy + 4 * name_cross_entropy. Name loss rewards probability assigned to same-name printings; exact-product loss retains printing supervision. Batches include same-name siblings and visually similar wrong-name negatives. Training uses augmented catalog images, never the evaluation scans. This does not add OCR or guarantee correct text recognition.

Evaluation

Ten production mismatch scans, replayed through production preprocessing:

The two wrong-name cases were Great Ball and Hau. Their exact printings are absent from the gallery; this release retrieves other Great Ball/Hau printings, with cosine similarities 0.614 and 0.704. Its top ten candidates for each scan all have the corresponding correct name.

On 514 historical Pokémon scans, exact-print top-1 is 508/514 (98.83%), compared with 508/514 for the prior CSR model and 509/514 for the training starting model. These historical scans have been used for model selection. The ten challenge scans are Pokémon-only and preselected failures; neither result establishes accuracy on arbitrary inputs or other games. No new latency benchmark is claimed.

A min_similarity=0.60 floor preserves all ten challenge predictions and rejects none of the 514 historical scans for this release. Cosine similarity is not a calibrated probability; evaluate rejection behavior on your own capture domain. Printing/finish variants with identical catalog images remain ambiguous.

Files and compatibility

• adapter_config.json, adapter_model.safetensors: PEFT vision-tower adapter.
• embeddings.pt: ids, L2-normalized fp16 embeds (57,913 × 1,152), aspect_ratios, and build provenance.
• evaluation.json: aggregate results, limitations, and artifact SHA-256 values.

Always deploy the adapter and gallery from the same revision. Existing embeddings from earlier adapters must not be mixed with these weights.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel
from huggingface_hub import hf_hub_download

repo = "jackttv/card-scanner-siglip-lora"
revision = "name-first-2026-10-12"  # pin both downloads to this release
base = "google/siglip2-so400m-patch14-384"
base_revision = "e8e487298228002f3d8a82e0cd5c8ea9c567f57f"
model = AutoModel.from_pretrained(base, revision=base_revision, dtype=torch.bfloat16)
model.vision_model = PeftModel.from_pretrained(model.vision_model, repo, revision=revision)
model.vision_model = model.vision_model.merge_and_unload()
model.eval()
processor = AutoProcessor.from_pretrained(base, revision=base_revision)
gallery = torch.load(hf_hub_download(repo, "embeddings.pt", revision=revision),
                     map_location="cpu", weights_only=True)

Production /identify decodes with OpenCV, resizes to 400×558 with INTER_LINEAR, converts BGR to RGB, then applies the processor. Gallery references are encoded directly with the processor. Query and gallery embeddings are L2-normalized and searched using fp16 cosine similarity.

The preceding Hub revision is retained under pre-name-first-2026-10-12.