MangaliCa

🐗 MangaliCa: A Bilingual Vision–Language Model for Hungarian–English Image Captioning and Retrieval

<div align="center"> <img src="./assets/mangalica.jpg" width="50%" /> </div>

Model Description

MangaliCa is the first publicly available Hungarian–English bilingual vision–language model designed for image captioning and image–text retrieval.
The model is built on the CoCa (Contrastive Captioner) framework and jointly optimizes contrastive alignment and autoregressive caption generation across two languages.

MangaliCa integrates:

• CLIP ViT-L/14 as a frozen vision encoder
• TinyLlama 1.1B as a text decoder, extended with cross-attention
• Language-conditioning tokens for explicit bilingual control
• LoRA adapters for memory-efficient fine-tuning

[!CAUTION] The model was trained on a newly constructed 70M-sample Hungarian–English bilingual image–caption dataset, the largest multimodal dataset involving Hungarian to date.

---

Intended Uses

Supported Tasks

• Image captioning (Hungarian and English)
• Image–text retrieval (text-to-image, image-to-text)
• Cross-lingual text–text retrieval
• Multimodal embedding extraction for RAG or search systems

Example Use Cases

• Hungarian-language multimodal assistants
• Bilingual image search engines
• Vision-enabled Retrieval-Augmented Generation (RAG)
• Research on medium-resource multilingual vision–language learning

---

🪛 Model Architecture

• Vision Encoder: CLIP ViT-L/14 (frozen, with LoRA adapters)
• Text Backbone: TinyLlama 1.1B
• Multimodal Fusion: Cross-attention layers in the decoder
• Training Objective:
  • Symmetric InfoNCE contrastive loss
  • Autoregressive captioning loss (CoCa-style)

Total parameters: ~1.8B
Trainable parameters (LoRA): ~15M

---

📋 Training Details

• Training data: ~11.2M bilingual samples (subset of a 70M dataset)
• Languages: Hungarian (hu), English (en)
• Hardware: NVIDIA T4 GPUs
• Optimization:
  • Adafactor optimizer
  • FP16/BF16 mixed precision
  • Gradient checkpointing
  • NF4 quantization
  • LoRA fine-tuning

---

📊 Evaluation

MangaliCa was evaluated on multiple benchmarks with Hungarian translations:

• GBC10M
• MS-COCO
• text-to-image-2M
• XM3600 (gold-standard Hungarian)

📌 Highlights

• Strong bilingual image–text retrieval performance
• Excellent English–Hungarian cross-lingual alignment
• Competitive captioning quality, especially on long-form captions

---

⚠️ Limitations

• The model is primarily optimized for Hungarian–English; performance may degrade on other languages.
• Training data is synthetic (machine-translated) and may contain translation artifacts.
• Not intended for sensitive, safety-critical, or high-stakes applications without further validation.