Pixel-Linguist-II-Midtrain

PIXEL LINGUIST II — Stage 1 + 2 (Pretraining + Semantic Mid-Training)

A vision-only encoder that reads, retrieves and compresses language directly in pixel space. Text is rendered to RGB images and encoded by the same tower that encodes natural images, so no text tokenizer is involved at inference time.

This is the main released checkpoint — the full two-stage curriculum (280M examples seen). It corresponds to the paper's pre-training + mid-training row.

Related releases:

From the EMNLP paper On the Design Fundamentals of Pixel Text Representation Learning.

Model details

Adjacent visual tokens are compressed by a 2x2 pooling layer; embeddings are mean-pooled over the (H/2)x(W/2) merged patches per image and L2-normalised.

Results

Spearman correlation for Visual STS, nDCG@5 for ViDoRe. Qwen2.5-ViT is the untrained backbone this model starts from.

Mid-training is what activates cross-lingual alignment (+9.1 over Stage 1) and document retrieval (+24.7) — the paper's RQ4 finding that large-scale unsupervised multilingual pretraining builds perceptual capability which is then activated by semantic mid-training.

Applying a further ~270K AllNLI triplet finetuning stage on top raises English Visual STS to ~80. That checkpoint is not released here.

Usage

import json, torch, torch.nn.functional as F
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoImageProcessor
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VisionTransformerPretrainedModel
from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig

path = snapshot_download("Pixel-Linguist/Pixel-Linguist-II-Midtrain")

config = Qwen2_5_VLVisionConfig(**json.load(open(f"{path}/config.json")))
model = Qwen2_5_VisionTransformerPretrainedModel(config)
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model = model.to("cuda", torch.bfloat16).eval()
processor = AutoImageProcessor.from_pretrained(path)


@torch.no_grad()
def encode(images):
    """Embed a list of PIL images (rendered text or natural photos)."""
    inputs = processor(images=images, return_tensors="pt")
    pixel_values = inputs["pixel_values"].to("cuda", torch.bfloat16)
    grid_thw = inputs["image_grid_thw"].to("cuda")

    hidden = model(hidden_states=pixel_values, grid_thw=grid_thw)

    sizes = ((grid_thw[:, 1] // 2) * (grid_thw[:, 2] // 2)).long()
    if sizes.sum() != hidden.shape[0]:
        sizes[-1] += hidden.shape[0] - sizes.sum()
    index = torch.repeat_interleave(torch.arange(len(grid_thw), device="cuda"), sizes)

    pooled = torch.zeros((len(grid_thw), hidden.shape[-1]), dtype=hidden.dtype, device="cuda")
    pooled.index_add_(0, index, hidden)
    pooled = pooled / sizes.unsqueeze(1).to(hidden.dtype).clamp(min=1)
    return F.normalize(pooled, dim=-1)

To embed a string, render it to an image first — that is the whole point of the model. Any renderer works; the paper samples from 393 fonts and 5,000+ DTD textured backgrounds during training so the encoder is robust to layout:

from PIL import Image, ImageDraw

def render(text, size=(448, 64)):
    img = Image.new("RGB", size, "white")
    ImageDraw.Draw(img).text((5, 25), text, fill="black")
    return img

emb = encode([render("a dog runs in the park"),
              render("a puppy is running outside"),
              render("quantum field theory")])
emb @ emb.T   # 0.68 for the paraphrase pair, 0.06 for the unrelated one

Reproducibility notes. transformers>=4.57 loads the image processor as Qwen2VLImageProcessorFast by default and warns that this "may produce slightly different outputs"; pass use_fast=False to AutoImageProcessor.from_pretrained if you need the slow processor. Scores are also sensitive to how text is rendered — the canvas size and font size act as spatial proxies for this model, so the same string rendered at a different width will give a different embedding.

For visual document retrieval, render the query as an image too and encode queries and page images through this same single tower.

Training

Two stages, same script and objective, differing only in the text corpus.

Stage 1: (62M wiki  + 26M LAION) x 2 epochs = 176M
Stage 2: (26M pairs + 26M LAION) x 2 epochs = 104M
                                      total = 280M examples seen

Objective is a symmetric in-batch InfoNCE with embeddings all-gathered across ranks (logit_scale = 1/0.03), on 64 GPUs with bf16, DeepSpeed ZeRO-2 and gradient checkpointing.

Text is rendered on the fly with randomised fonts, sizes, backgrounds, brightness and blur, so the model never sees the same visual instantiation of a string twice. This suppresses the pixel-level shortcut learning documented in the paper's RQ3. Natural image-text pairs act as a required regulariser — dropping them collapses document retrieval even at full data scale (RQ2).

Optical context compression

Because representations are compact, the model tolerates aggressive visual token downsampling: it matches CLIP on Visual STS with 60% of tokens discarded, and still beats uncompressed CLIP on ViDoRe at 80% compression — useful for using the visual modality as a context-compression medium.

Citation

@inproceedings{yuan2026pixel,
  title     = {On the Design Fundamentals of Pixel Text Representation Learning},
  author    = {Yuan, Chaohao and Yuan, Ruifeng and Huang, Zhuoxu and Rong, Yu and
               Cheng, Hong and Chan, Hou Pong and Xiao, Chenghao},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}