aya-vision-32b-americas

Model Card for ludolara/aya-vision-32b-americas

ludolara/aya-vision-32b-americas is a LoRA adapter for Aya Vision 32B trained as the machine-translation stage (SFT (MT)) of the AmericasNLP 2026 system pipeline. It adapts the base VLM to generate Spanish and Indigenous-language text for Bribri, Guarani, Nahuatl, and Wixárika.

Model Details

Model Description

• Developed by: Mila - Quebec AI Institute (Luis Lara, Param Raval)
• Funded by [optional]: Not explicitly reported
• Shared by [optional]: Luis Lara (ludolara)
• Model type: PEFT LoRA adapter for a multimodal causal language model (Aya Vision 32B)
• Language(s) (NLP): Spanish (es), Bribri (bzd), Guarani (grn/gn), Nahuatl (nah), Wixárika (hch)
• License: cc-by-nc-4.0 (inherits base-model license and usage constraints)
• Finetuned from model [optional]: CohereLabs/aya-vision-32b

Model Sources [optional]

• Repository: https://github.com/ludolara/americasnlp2026
• Paper [optional]: From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas (AmericasNLP 2026 system description)
• Demo [optional]: Not provided

Uses

Direct Use

• Spanish-to-Indigenous and Indigenous-to-Spanish machine translation candidate generation for the four overlapping AmericasNLP language tracks.
• Intermediate adapter checkpoint before downstream image-captioning fine-tuning.

Downstream Use [optional]

• Initialization checkpoint for SFT (IC) captioning training.
• Base checkpoint for optional RLVR (MT) / GRPO refinement.

Out-of-Scope Use

• Safety-critical or professional translation without expert review.
• Deployment as a culturally authoritative system without community oversight.
• Claims of parity with certified human translators for legal, medical, or educational contexts.

Bias, Risks, and Limitations

• Training and evaluation focus on low-resource settings with limited supervised data.
• Reported translation results use reference-based best-of-100 candidate selection, which is not equivalent to real interactive translation.
• Outputs may contain factual errors, harmful stereotypes, or culturally inappropriate wording.
• The model can underperform on dialectal variation or domains not represented in the shared-task resources.

Recommendations

• Keep a human-in-the-loop, ideally with fluent speakers and community experts.
• Treat reported metrics as candidate-quality signals, not production readiness.
• Add task-specific moderation and output validation before any real-world deployment.

How to Get Started with the Model

Use the code below to run this adapter on top of Aya Vision 32B.

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base_model_id = "CohereLabs/aya-vision-32b"
adapter_id = "ludolara/aya-vision-32b-americas"

processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    base_model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

prompt = "Traduce del español al wixárika. La casa está lejos del pueblo."
messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Training Details

Training Data

Machine-translation stage (SFT (MT)) uses Spanish-Indigenous parallel resources for:

• Bribri (bzd)
• Guarani (grn)
• Nahuatl (nah)
• Wixárika (hch)

The system description references resources including Axolotl (Spanish-Nahuatl), Guarani-Spanish corpora, a Bribri back-translation resource, a Wixárika resource, and additional AmericasNLP shared-task materials.

Training Procedure

Preprocessing [optional]

• Data is formatted as instruction-style chat prompts.
• Training is bidirectional (Spanish->Indigenous and Indigenous->Spanish).
• Target text is used as assistant response.

Training Hyperparameters

• Training regime: LoRA SFT with bf16 + DeepSpeed ZeRO-3, early stopping enabled
• Key config values (configs/lora_sft.yaml):
  • epochs: 60 (early-stopped around 6.8 epochs in paper)
  • learning rate: 2e-5
  • warmup ratio: 0.03
  • weight decay: 0.01
  • per-device batch size: 32
  • gradient accumulation: 1
  • max sequence length: 256
  • LoRA rank/alpha/dropout: 32 / 64 / 0.1

Speeds, Sizes, Times [optional]

• Hardware budget in scripts targets 4 x H100 GPUs on a Slurm cluster.
• End-to-end wall-clock training time is not explicitly reported.

Evaluation

Testing Data, Factors & Metrics

Testing Data

• AmericasNLP MT shared-task test split (as used in the system description for overlapping languages).

Factors

• Language (bzd, grn, hch, nah)
• Translation direction (SPA->XXX, XXX->SPA)

Metrics

• chrF++ (Popovic, 2017)
• Candidate selection setting: best-of-100 with reference-based sentence-level chrF++

Results

Reported for this stage (SFT (MT)) in the paper:

Summary

This adapter establishes the translation-capable initialization used by later captioning models. It is intended as an intermediate checkpoint in the multi-stage pipeline.

Model Examination [optional]

No dedicated interpretability analysis was reported.

Environmental Impact

Carbon emissions can be estimated with the Machine Learning Impact calculator.

• Hardware Type: NVIDIA H100 GPUs (script target: 4 GPUs)
• Hours used: Not reported
• Cloud Provider: Not reported (training scripts are Slurm/HPC-oriented)
• Compute Region: Not reported
• Carbon Emitted: Not reported

Technical Specifications [optional]

Model Architecture and Objective

• Base architecture: Aya Vision 32B multimodal autoregressive model
• Adaptation method: LoRA on attention and MLP projection modules (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• Objective: Supervised next-token training for translation prompts

Compute Infrastructure

DeepSpeed ZeRO-3 distributed setup via Slurm + Accelerate wrappers.

Hardware

• Multi-GPU H100 configuration in training scripts
• 128 GB host memory target in scripts

Software

• torch>=2.2.0
• transformers>=4.46.0
• datasets>=2.21.0
• accelerate>=0.34.0
• deepspeed>=0.16.0
• trl>=0.11.0
• peft 0.18.1
• huggingface_hub>=0.26.0

Citation [optional]

BibTeX:

@inproceedings{lara-americasnlp-2026,
  title = "From Machine Translation to Image Captioning: Training Vision-Language Models for {I}ndigenous Languages of the {A}mericas",
  author = {Lara, Luis and
  Raval, Param},
  booktitle = "Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)",
  month = jul,
  year = "2026",
  address = "San Diego, California",
  publisher = "Association for Computational Linguistics",
}

APA:

Lara, L., & Raval, P. (2026). From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas. AmericasNLP 2026.

Glossary [optional]

• SFT (MT): Supervised fine-tuning on machine-translation data.
• LoRA: Low-Rank Adaptation.
• chrF++: Character n-gram MT metric with word-order extension.

More Information [optional]

This checkpoint is part of a staged research pipeline. For final shared-task captioning submissions, see the captioning adapters in the same namespace.

Model Card Authors [optional]

Luis Lara (maintainer), with model-card synthesis from project documentation and system paper.

Model Card Contact

[email protected]

Framework versions

• PEFT 0.18.1