Flantier2-SmolVLM-2B-dse

Flantier2-SmolVLM-2B-dse

A lightweight multimodal vision-language model specialized for multilingual technical document retrieval.

Overview

Flantier2-SmolVLM-2B-dse is the second generation of the Document Screenshot Embedding model, a 2B parameter vision-language model designed for efficient retrieval of technical documentation. It directly encodes document screenshots into embeddings, preserving all information including text, images, and layout without requiring separate content extraction.

Performance

ENERGY Benchmark (racineai/Open-VLM-Retrieval-Leaderboard)

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6787ab9c76a88637df40fb7a/1nOC2d_6yt7QtNzi2dR-V.png" alt="Flantier Comparison" width="800"> </p>

Key Strengths of Flantier2

• Significant multilingual improvement: +3.25% relative improvement achieving 0.8197 average (vs 0.7939 for Flantier1)
• Excellent performance across all European languages:
  • Strong performance in English (0.8777)
  • Major improvement in Italian (+6.85%) and German (+5.73%)
  • Better balance across all tested languages
• Efficient training: Trained on approximately 2% of the OGC_MEGA dataset

Training Strategy and Results

Flantier2 was fine-tuned on a carefully curated subset of the OGC_MEGA_Document retrieval dataset, representing approximately 2% of the full dataset. The training strategy leveraged hard negatives and the dataset's linguistic diversity to maximize data efficiency.

This approach enabled the model to learn robust cross-lingual features, successfully rebalancing language representation. The strategic rebalancing resulted in a slight -1.03% decrease in English (0.8868 → 0.8777) but achieved substantial improvements in underperforming languages: Italian (+6.85%), German (+5.73%), French (+3.88%), and Spanish (+1.67%), resulting in a +3.25% overall relative improvement (0.8197 vs 0.7939 average).

Multilingual Balance Improvement

Flantier2 demonstrates significantly better multilingual balance compared to Flantier1, as measured by statistical dispersion metrics across the five supported languages:

• Variance: 42.4% reduction (0.0036 → 0.0021) - shows more consistent performance across languages
• Standard Deviation: 24.1% reduction (0.0602 → 0.0457) - indicates tighter clustering of language scores
• Coefficient of Variation: 26.5% reduction (7.59% → 5.58%) - demonstrates improved relative consistency
• Score Range: 27.9% reduction (0.1782 → 0.1285) - reflects a narrower gap between best and worst performing languages

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6787ab9c76a88637df40fb7a/cDx-tsiPI4tb3ckJ4ZQ6k.png" alt="Multilingual Balance Metrics" width="800"> </p>

These metrics confirm that Flantier2 achieves approximately 24-42% better balance across all five European languages, successfully reducing the performance disparities observed in the first generation while maintaining high absolute scores.

Comparison with Other Models

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6787ab9c76a88637df40fb7a/4oemnFzmjiTwJvh_DEVge.png" alt="ENERGY Benchmark Comparison" width="800"> </p>

Key Distinction: Flantier2 is specifically optimized for balanced European multilingual performance (EN, FR, DE, ES, IT). While achieving an average score of 0.8197, the model prioritizes uniform language coverage and training efficiency (≈2% of dataset) over peak performance. This strategic trade-off resulted in a slight decrease in English (-1.03%) to achieve substantial improvements across other European languages (+1.67% to +6.85%), demonstrating better cross-lingual consistency.

Key Features

• Efficient Retrieval: Generates document and query embeddings for semantic similarity search
• Multimodal Understanding: Processes text, diagrams, charts, and tables in their original layout
• Lightweight Architecture: Only 2B parameters, runs on consumer GPUs
• Enhanced Multilingual Support: Optimized performance on English, French, German, Spanish, and Italian
• No Preprocessing Required: Directly works with document screenshots

Installation

pip install transformers accelerate pillow

Usage

from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq

# Load model and processor
processor = AutoProcessor.from_pretrained("racineai/Flantier2-SmolVLM-2B-dse")
model = AutoModelForVision2Seq.from_pretrained(
    "racineai/Flantier2-SmolVLM-2B-dse",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load document image
document_image = Image.open("technical_document.jpg")

# Process for document embedding
doc_messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"}
        ]
    },
]
doc_prompt = processor.apply_chat_template(doc_messages, add_generation_prompt=True)
doc_inputs = processor(text=doc_prompt, images=[document_image], return_tensors="pt").to(model.device)

# Generate document embedding
with torch.no_grad():
    doc_outputs = model(**doc_inputs, output_hidden_states=True, return_dict=True)
    doc_embedding = doc_outputs.hidden_states[-1][:, -1]  # Last token embedding
    doc_embedding = torch.nn.functional.normalize(doc_embedding, p=2, dim=-1)

# Process query embedding
query = "What are the specifications of this component?"
query_messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": query}
        ]
    },
]
query_prompt = processor.apply_chat_template(query_messages, add_generation_prompt=True)
query_inputs = processor(text=query_prompt, return_tensors="pt").to(model.device)

# Generate query embedding
with torch.no_grad():
    query_outputs = model(**query_inputs, output_hidden_states=True, return_dict=True)
    query_embedding = query_outputs.hidden_states[-1][:, -1]  # Last token embedding
    query_embedding = torch.nn.functional.normalize(query_embedding, p=2, dim=-1)

# Calculate similarity
similarity = torch.nn.functional.cosine_similarity(query_embedding, doc_embedding)
print(f"Similarity score: {similarity.item():.4f}")

Applications

• Multilingual Technical Document Retrieval: Find relevant documents across multiple European languages
• International Technical Support Systems: Match user questions to relevant documentation regardless of language
• Engineering Knowledge Management: Index and search technical specifications, diagrams, and reports
• European Technical Documentation: Optimized support for documents in EN, FR, DE, ES, and IT

Training Methodology

Flantier2 was trained using the improved Document Screenshot Embedding (DSE) approach, which treats document screenshots as a unified input format. This second generation was developed with remarkable efficiency:

• OGC_MEGA Dataset: Trained on approximately 2% of the complete dataset, demonstrating excellent learning efficiency
• Enriched Multilingual Data: Strategic selection of examples covering 5 European languages
• Cross-lingual Optimization: Better balance of performance across languages
• Refined Architecture: Improvements for multimodal and multilingual understanding

This eliminates the need for content extraction preprocessing while preserving all visual and textual information in documents.

Training Pipeline

Architecture

SmolVLM-Instruct serves as base model with LoRA adaptation for efficient fine-tuning. The system uses BFloat16 precision, Flash Attention 2, and last-token pooling for representation extraction.

Dataset

OGC_MEGA comprises 1.09M examples across five languages. Each example contains a query, one positive document image, and up to 16 hard negatives. Critically, 25% include hard negatives while 75% do not.

Negative Sampling

Three complementary sources provide negatives: dataset hard negatives (semantically similar but incorrect), in-batch negatives (other queries' positives), and distributed negatives (embeddings from all GPUs in multi-GPU training).

Adaptive Batching

Two batch types address dataset heterogeneity: standard-sized batches for examples with hard negatives, and enlarged batches for examples without hard negatives to maintain constant passage volume through increased in-batch negatives.

Training Process

Data follows SmolVLM chat format. The InfoNCE loss treats similarity matrices as classification tasks, maximizing query-positive similarity while minimizing query-negative similarity. DeepSpeed ZeRO Stage 3 enables distributed training with AdamW optimization, linear warmup (10% of steps), and gradient checkpointing.

Outcome

This architecture achieves efficient multilingual visual document retrieval through parametric adaptation, multi-source negative exploitation, adaptive batching, and distributed optimization on accessible hardware.

What's New in Flantier2

• +3.25% relative improvement on the ENERGY benchmark (0.8197 vs 0.7939 average)
• Major progress on Italian (+6.85%) and German (+5.73%)
• Better overall multilingual balance (24-42% reduction in dispersion metrics)
• Strategic rebalancing: slight English decrease (-1.03%) for substantial gains in other European languages
• Ultra-efficient training on ≈2% of the OGC_MEGA dataset

Authors

Léo Appourchaux - Lead Developer at TW3 Partners

Noé Brandolini - R&D at TW3 Partners - Student at École Centrale d'Électronique

Paul Lemaistre - GD at Racine.ai – Adjunct Professor at École Centrale d'Électronique

Citation

@misc{flantier2-smolvlm-2b-dse,
  title={Flantier2-SmolVLM-2B-dse: A Lightweight Multimodal Vision-Language Model for Multilingual Technical Document Retrieval},
  author={Racine AI},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/racineai/Flantier2-SmolVLM-2B-dse}},
  note={Fine-tuned from HuggingFaceTB/SmolVLM-Instruct on OGC\_MEGA\_MultiDomain\_DocRetrieval dataset}
}

@dataset{ogc_mega_dataset,
  title={OGC\_MEGA\_MultiDomain\_DocRetrieval: A Multilingual Document Retrieval Dataset},
  author={Racine AI},
  year={2025},
  publisher={HuggingFace Datasets},
  howpublished={\url{https://huggingface.co/datasets/racineai/OGC_MEGA_MultiDomain_DocRetrieval}}
}