Downloads · 30 days
28
100% of all-time downloads
ukiyoe-haiku-vlm-en
MR0b0t/ukiyoe-haiku-vlm-en
ukiyoe-haiku-vlm-en is a machine learning model from MR0b0t. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A multimodal model that generates 5-7-5 English haiku from ukiyo-e woodblock prints. Built from scratch as a LLaVA-pattern VLM: SigLIP-base vision encoder + custom 2-layer MLP projector + Qwen2.5-3B-Instruct + LoRA ad…
All-time downloads
28
Public
Repo size
1.4 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.4 GB · 98%
At a glance
- License
- apache-2.0
- Model type
- ukiyoe-haiku-vlm
- Access
- Public
- Created
- May 23, 2026
- Updated
- Nov 2, 2026
- SHA
- cfccca8e
Base models
Model card
From the Hugging Face model README
Ukiyo-e Haiku VLM (English)
A multimodal model that generates 5-7-5 English haiku from ukiyo-e woodblock prints. Built from scratch as a LLaVA-pattern VLM: SigLIP-base vision encoder + custom 2-layer MLP projector + Qwen2.5-3B-Instruct + LoRA adapter.
This is the SFT checkpoint — trained on synthetic haiku captions from Claude Haiku 4.5. The companion ukiyoe-haiku-vlm-jp model adds preference optimization on top; for English, SFT alone produced the best results in our ablation.
Quick start
pip install torch transformers peft pillow accelerate
# Download repo
huggingface-cli download MR0b0t/ukiyoe-haiku-vlm-en --local-dir ./ukiyoe-vlm-en
# Run inference
cd ./ukiyoe-vlm-en
python inference.py path/to/ukiyoe.jpg --lang en
Architecture
| Component | Details |
|---|---|
| Vision encoder | google/siglip-base-patch16-224 (frozen, 86M params) |
| Projector | 2-layer MLP (768→2048→2048, GELU), trained from scratch (~3.5M) |
| Language model | Qwen/Qwen2.5-3B-Instruct (frozen) |
| Adapter | LoRA r=16 on q/k/v/o + gate/up/down (~32M trainable) |
| Total | 3.21B params, ~36M trainable (1.1%) |
Training
Trained on 3713 image-haiku pairs (200 held-out) from 3913 Met Museum CC0 ukiyo-e prints. 3 epochs of SFT, batch size 8, lr=2e-4, ~40 min on a single A100 40GB.
Captions generated by Claude Haiku 4.5 with two prompts:
- Chosen (used for training): rich, metadata-grounded prompt
- Rejected (used for preference experiments below): bare-bones prompt
Evaluation
Held-out 200-image test set, scored by Claude Haiku 4.5 as judge.
| Model | structure | image_fit | poetic | cultural | avg |
|---|---|---|---|---|---|
| Random init (no train) | 3.24 | 1.56 | 3.43 | 3.06 | 2.82 |
| Blind Qwen (no image) | 4.19 | 1.69 | 3.98 | 3.25 | 3.28 |
| SFT (this model) | 4.66 | 3.80 | 3.94 | 4.31 | 4.18 |
| ORPO (same-model contrast) | 4.60 | 3.84 | 3.90 | 4.30 | 4.16 |
| KTO (same-model contrast) | 4.62 | 3.87 | 3.92 | 4.26 | 4.17 |
Image-fit lifted from 1.69 (blind baseline) to 3.80 (this model) — confirming the vision pipeline learned to ground captions in image content.
ORPO and KTO did NOT improve over SFT in this English run because the chosen and rejected captions both came from the same model (Claude Haiku 4.5) with only prompt variation — insufficient quality contrast for preference learning. The Japanese variant (ukiyoe-haiku-vlm-jp) addresses this by using model-level contrast (Sonnet vs Haiku) and shows real ORPO gains.
Caveats
- Specialized model. Trained only on ukiyo-e prints.
- Not a Qwen-VL competitor. Educational/research project.
- Judge bias floor. Same-family caption + judge introduces small bias in absolute scores.
Reproducibility
Full training code: https://github.com/debtirthasaha/ukiyoe-haiku-vlm
License
Apache 2.0 for code and adapter weights. Base models retain their original licenses.
- Type
- ukiyoe-haiku-vlm
- License
- apache-2.0
- Languages
- en
- Created
- May 23, 2026
- Updated
- Nov 2, 2026