minicpm-v4_6-captcha-lora

MiniCPM-V-4.6 CAPTCHA Solver (QLoRA adapter)

QLoRA adapter fine-tuning openbmb/MiniCPM-V-4.6 to read distorted alphanumeric text from synthetic CAPTCHA images and output the exact character sequence.

This release is the noise-robust checkpoint: it was trained from scratch on data with on-the-fly noise augmentation (salt & pepper, Gaussian, speckle, blur, JPEG artifacts, and variable-scale pixelation), which improves accuracy on corrupted / low-quality inputs while matching the clean-image baseline.

Results

Greedy decoding, 500-sample held-out validation split.

For comparison, the previous (non-augmented) adapter scored 41.8% / 69.6% on the same noisy set — this checkpoint is +3.8 pts case-sensitive and +4.0 pts case-insensitive more robust to noise.

Note on the case-sensitive vs case-insensitive gap. The ~30-point gap is almost entirely case ambiguity: for glyphs like c/C, o/O, s/S, u/U, v/V, w/W, x/X, z/Z the upper- and lower-case forms are visually identical apart from size, so case is inherently hard to recover from a single distorted glyph. Reading the characters themselves is ~85% reliable.

Model details

• Base model: openbmb/MiniCPM-V-4.6 (vision-language, ~1.31B params total)
• Method: QLoRA (4-bit NF4 base + double quant, bf16 compute)
• LoRA: r=16, alpha=32, dropout=0.05; targets q/k/v/o_proj, gate/up/down_proj
• Trainable params: 9,486,336 (0.72%)
• Vision encoder: frozen

Training

• Data: synthetic CAPTCHAs (4–7 chars, [a-zA-Z0-9] minus ambiguous I O 0 1 l), 10k train / 2k test; multi-font, warping, lines, background noise. See the companion dataset: Hansda/captcha-dataset.
• Augmentation (train only): salt & pepper, Gaussian, speckle, Gaussian blur, JPEG artifacts, downscale→upscale pixelation (random scale); val/test kept clean.
• Hardware: single NVIDIA RTX 5060 Ti (16 GB)
• Optimizer: paged_adamw_8bit, LR 2e-4 cosine, effective batch 32, 4 epochs
• Precision: bf16, gradient checkpointing

Prompt format

Single-turn, one image + instruction:

What is the text shown in this CAPTCHA image? Reply with only the exact characters, nothing else.

Usage

import torch
from transformers import AutoProcessor, BitsAndBytesConfig
from transformers.models.minicpmv4_6.modeling_minicpmv4_6 import MiniCPMV4_6ForConditionalGeneration
from peft import PeftModel
from PIL import Image

model_id = "openbmb/MiniCPM-V-4.6"
adapter  = "Hansda/minicpm-v4_6-captcha-lora"

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
base = MiniCPMV4_6ForConditionalGeneration.from_pretrained(
    model_id, quantization_config=bnb, dtype=torch.bfloat16,
    device_map="auto", attn_implementation="eager")
model = PeftModel.from_pretrained(base, adapter).eval()
proc  = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

img = Image.open("captcha.png").convert("RGB")
instr = "What is the text shown in this CAPTCHA image? Reply with only the exact characters, nothing else."
text = proc.apply_chat_template(
    [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": instr}]}],
    tokenize=False, add_generation_prompt=True)
inp = proc(images=[img], text=[text], return_tensors="pt").to(model.device)
if "pixel_values" in inp: inp["pixel_values"] = inp["pixel_values"].to(torch.bfloat16)
out = model.generate(**inp, max_new_tokens=32, do_sample=False,
                     pad_token_id=proc.tokenizer.pad_token_id)
print(proc.tokenizer.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Limitations

• Trained purely on synthetic CAPTCHAs; performance on a specific real-world CAPTCHA style depends on how closely it matches the training distribution.
• Case-sensitive accuracy is bounded by glyph case ambiguity (see note above).
• Intended for research / accessibility / dataset-labelling use.