Downloads · 30 days
15
17% of all-time downloads
ludolara/aya-vision-32b-americas-grpo-captioning
aya-vision-32b-americas-grpo-captioning is a image-text-to-text model from ludolara. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as cc-by-nc-4.0.
ludolara/aya-vision-32b-americas-grpo-captioning is a LoRA adapter for Aya Vision 32B trained with a three-stage pipeline:
Downloads · 30 days
15
17% of all-time downloads
All-time downloads
86
Public
Repo size
415 MB
Likes
0
Public
Click a slice to open those files.
.safetensors395 MB · 95%
From the Hugging Face model README
ludolara/aya-vision-32b-americas-grpo-captioningludolara/aya-vision-32b-americas-grpo-captioning is a LoRA adapter for Aya Vision 32B trained with a three-stage pipeline:
SFT (MT) on Spanish-Indigenous translation data.RLVR (MT) using GRPO with chrF++-based rewards.SFT (IC) for direct Indigenous-language image captioning.This is the GRPO-enhanced captioning branch reported in the AmericasNLP 2026 system description.
ludolara)bzd), Guarani (grn/gn), Nahuatl (nah), Wixárika (hch)cc-by-nc-4.0 (inherits base-model license and usage constraints)CohereLabs/aya-vision-32bSFT (MT) + RLVR (MT) + SFT (IC) in low-resource multilingual captioning.Use the code below to run image captioning with this adapter.
import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base_model_id = "CohereLabs/aya-vision-32b"
adapter_id = "ludolara/aya-vision-32b-americas-grpo-captioning"
processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
base_model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()
image = Image.open("example.jpg").convert("RGB")
language = "guarani"
prompt = (
f"Escribe un solo pie de foto en {language} para esta imagen. "
f"Debe ser una descripcion culturalmente adecuada de la imagen. "
f"Responde solo con el pie de foto en {language}, sin explicaciones."
)
messages = [{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": prompt},
],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))
This adapter uses three stages:
SFT (MT)): Spanish-Indigenous translation data for bzd, grn, nah, hch.RLVR (MT)): Same MT data with GRPO optimization and chrF++ reward.SFT (IC)): AmericasNLP 2026 captioning development data for Bribri, Guarani, Nahuatl, Wixárika.configs/lora_grpo.yaml) highlights:
301e-60.032080.92564000.3configs/captioning_lora_sft.yaml) highlights:
102e-5140964 x H100 GPUs on Slurm.bribri, guarani, nahuatl, wixarika)Official test chrF++ (paper Table 1, GRPO variant):
| Version | Submission | Overall | Bribri | Guarani | Nahuatl | Wixárika |
|---|---|---|---|---|---|---|
| v1 | SFT (MT) + RLVR (MT) + SFT (IC) | 17.341 | 10.886 | 19.772 | 19.853 | 18.853 |
Internal dev subset chrF++ (paper Table 2):
| System | Overall | Bribri | Guarani | Nahuatl | Wixárika |
|---|---|---|---|---|---|
| SFT (MT) + RLVR (MT) + SFT (IC) | 19.805 | 11.508 | 23.767 | 16.982 | 22.496 |
Human evaluation for this branch (paper Table 3):
v1: rank 5, mean rating 1.764In this setup, GRPO provided a small dev-set gain versus plain SFT (MT) + SFT (IC), but did not consistently improve all languages on the official test set.
No dedicated interpretability analysis was reported.
Carbon emissions can be estimated with the Machine Learning Impact calculator.
DeepSpeed ZeRO-3 + Accelerate on Slurm.
torch>=2.2.0transformers>=4.46.0datasets>=2.21.0accelerate>=0.34.0deepspeed>=0.16.0trl>=0.11.0peft 0.18.1huggingface_hub>=0.26.0BibTeX:
@inproceedings{lara-americasnlp-2026,
title = "From Machine Translation to Image Captioning: Training Vision-Language Models for {I}ndigenous Languages of the {A}mericas",
author = {Lara, Luis and
Raval, Param},
booktitle = "Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)",
month = jul,
year = "2026",
address = "San Diego, California",
publisher = "Association for Computational Linguistics",
}
APA:
Lara, L., & Raval, P. (2026). From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas. AmericasNLP 2026.
This is the GRPO-enhanced captioning branch. For the non-GRPO captioning branch, see ludolara/aya-vision-32b-americas-captioning.
Luis Lara (maintainer), with model-card synthesis from project documentation and system paper.