aya-vision-32b-americas-captioning

Model Card for ludolara/aya-vision-32b-americas-captioning

ludolara/aya-vision-32b-americas-captioning is a LoRA adapter for Aya Vision 32B trained with a two-stage pipeline:

1. SFT (MT) on Spanish-Indigenous translation data.
2. SFT (IC) on AmericasNLP 2026 cultural image-captioning development data.

It is part of the Mila submission pipeline for the AmericasNLP 2026 shared task on cultural image captioning.

Model Details

Model Description

• Developed by: Mila - Quebec AI Institute (Luis Lara, Param Raval)
• Funded by [optional]: Not explicitly reported
• Shared by [optional]: Luis Lara (ludolara)
• Model type: PEFT LoRA adapter for Aya Vision 32B (multimodal causal language model)
• Language(s) (NLP): Bribri (bzd), Guarani (grn/gn), Nahuatl (nah), Wixárika (hch)
• License: cc-by-nc-4.0 (inherits base-model license and usage constraints)
• Finetuned from model [optional]: CohereLabs/aya-vision-32b

Model Sources [optional]

• Repository: https://github.com/ludolara/americasnlp2026
• Paper [optional]: From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas (AmericasNLP 2026 system description)
• Demo [optional]: Not provided

Uses

Direct Use

• Direct image captioning in the requested Indigenous language for Bribri, Guarani, Nahuatl, and Wixárika.
• Caption generation for AmericasNLP-style culturally grounded prompts.

Downstream Use [optional]

• Baseline captioning adapter for additional task-specific tuning or decoding experiments.
• Reference checkpoint family for the non-GRPO captioning submissions (SFT (MT) + SFT (IC)).

Out-of-Scope Use

• Fully automated cultural or educational deployment without speaker/community review.
• Safety-critical use where incorrect or culturally inappropriate text can cause harm.
• Use as a replacement for professional translators, educators, or community language authorities.

Bias, Risks, and Limitations

• Small-captioning-data regime (50 examples/language in dev) increases instability and overfitting risk.
• Automatic chrF++ is limited for cultural appropriateness evaluation.
• Language coverage is limited to four AmericasNLP tracks in this adapter.
• The model may generate fluent but culturally incorrect captions.

Recommendations

• Include expert human review (native speakers/community stakeholders) for real-world use.
• Evaluate both linguistic quality and cultural correctness, not just automatic metrics.
• Prefer conservative deployment settings and monitor outputs for harm.

How to Get Started with the Model

Use the code below to run image captioning with this adapter.

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base_model_id = "CohereLabs/aya-vision-32b"
adapter_id = "ludolara/aya-vision-32b-americas-captioning"

processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    base_model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

image = Image.open("example.jpg").convert("RGB")
language = "wixarika"
prompt = (
    f"Escribe un solo pie de foto en {language} para esta imagen. "
    f"Debe ser una descripcion culturalmente adecuada de la imagen. "
    f"Responde solo con el pie de foto en {language}, sin explicaciones."
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": prompt},
    ],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Training Details

Training Data

This adapter uses two data stages:

• Stage 1 (SFT (MT)): Spanish-Indigenous translation resources for bzd, grn, nah, hch.
• Stage 2 (SFT (IC)): AmericasNLP 2026 captioning development data (50 examples/language for Bribri, Guarani, Nahuatl, Wixárika).

Training Procedure

Preprocessing [optional]

• Translation stage uses instruction-style bilingual prompts and bidirectional labeling.
• Captioning stage formats each example with a single Spanish prompt requesting one culturally appropriate caption in the target language.
• Raw captioning dev split is used as validation split for SFT training in this codebase.

Training Hyperparameters

• Training regime: Multi-stage LoRA SFT with bf16 + DeepSpeed ZeRO-3
• Captioning-stage config (configs/captioning_lora_sft.yaml):
  • model init: outputs/aya-vision-32b-americas
  • epochs: 10 (paper reports selected checkpoints around epoch 9)
  • learning rate: 2e-5
  • warmup ratio: 0.03
  • weight decay: 0.01
  • per-device batch size: 1
  • gradient accumulation: 1
  • max sequence length: 4096
  • LoRA rank/alpha/dropout: 64 / 64 / 0.1
  • held-out eval fraction in config: 0.1

Speeds, Sizes, Times [optional]

• Training scripts target 4 x H100 GPUs on Slurm.
• End-to-end wall-clock timing is not explicitly reported.

Evaluation

Testing Data, Factors & Metrics

Testing Data

• AmericasNLP 2026 shared-task test set for Bribri, Guarani, Nahuatl, and Wixárika.
• Internal 20-example dev subset (5 examples per language, seed 42) for model selection.

Factors

• Language (bribri, guarani, nahuatl, wixarika)
• Pipeline variant / checkpoint version (v0, v2 in paper for this training recipe)

Metrics

• chrF++ for automatic ranking.
• Human-evaluation rank/score in official stage-2 evaluation.

Results

Official test chrF++ (paper Table 1, same recipe with different selected checkpoints):

Internal dev subset chrF++ (paper Table 2):

Human evaluation examples using this recipe family (paper Table 3):

• Bribri v0: rank 4, mean rating 1.994
• Nahuatl v2: rank 3, mean rating 1.560
• Wixárika v2: rank 5, mean rating 2.210

Summary

This adapter family is the strongest non-GRPO captioning route in the reported experiments and provides the main reproducible SFT (MT) + SFT (IC) baseline for the shared task.

Model Examination [optional]

No dedicated interpretability analysis was reported.

Environmental Impact

Carbon emissions can be estimated with the Machine Learning Impact calculator.

• Hardware Type: NVIDIA H100 GPUs (script target: 4 GPUs)
• Hours used: Not reported
• Cloud Provider: Not reported (Slurm/HPC setup)
• Compute Region: Not reported
• Carbon Emitted: Not reported

Technical Specifications [optional]

Model Architecture and Objective

• Base architecture: Aya Vision 32B multimodal autoregressive model
• Adaptation method: LoRA on core attention/MLP projection modules
• Objective: next-token prediction for bilingual text prompts (stage 1) and multimodal captioning prompts (stage 2)

Compute Infrastructure

DeepSpeed ZeRO-3 + Accelerate, launched through Slurm job scripts.

Hardware

• Multi-GPU H100 configuration in training scripts
• 128 GB host memory target in scripts

Software

• torch>=2.2.0
• transformers>=4.46.0
• datasets>=2.21.0
• accelerate>=0.34.0
• deepspeed>=0.16.0
• trl>=0.11.0
• peft 0.18.1
• huggingface_hub>=0.26.0

Citation [optional]

BibTeX:

@inproceedings{lara-americasnlp-2026,
  title = "From Machine Translation to Image Captioning: Training Vision-Language Models for {I}ndigenous Languages of the {A}mericas",
  author = {Lara, Luis and
  Raval, Param},
  booktitle = "Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)",
  month = jul,
  year = "2026",
  address = "San Diego, California",
  publisher = "Association for Computational Linguistics",
}

APA:

Lara, L., & Raval, P. (2026). From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas. AmericasNLP 2026.

Glossary [optional]

• SFT (MT): Supervised fine-tuning on machine-translation data.
• SFT (IC): Supervised fine-tuning on image-captioning data.
• chrF++: Character n-gram metric with word-order extension.

More Information [optional]

This repo contains the non-GRPO captioning adapter track. For the GRPO-enhanced variant, see ludolara/aya-vision-32b-americas-grpo-captioning.

Model Card Authors [optional]

Luis Lara (maintainer), with model-card synthesis from project documentation and system paper.

Model Card Contact

[email protected]

Framework versions

• PEFT 0.18.1