evoxtral

Evoxtral LoRA — Expressive Tagged Transcription

A LoRA adapter for Voxtral-Mini-3B-2507 that produces transcriptions enriched with inline expressive audio tags from the ElevenLabs v3 tag set.

Built for the Mistral AI Online Hackathon 2026 (W&B Fine-Tuning Track).

Two model variants available:

• Evoxtral SFT — Best overall transcription accuracy (lowest WER)
• Evoxtral RL — Best expressive tag accuracy (highest Tag F1)

What It Does

Standard ASR:

So I was thinking maybe we could try that new restaurant downtown. I mean if you're free this weekend.

Evoxtral:

[nervous] So... [stammers] I was thinking maybe we could... [clears throat] try that new restaurant downtown? [laughs nervously] I mean, if you're free this weekend?

Training Pipeline

Base Voxtral-Mini-3B → SFT (LoRA, 3 epochs) → RL (RAFT, 1 epoch)

1. SFT: LoRA finetuning on 808 synthetic audio samples with expressive tags (lr=2e-4, 3 epochs)
2. RL (RAFT): Rejection sampling — generate 4 completions per sample, score with rule-based reward (WER accuracy + Tag F1 - hallucination penalty), keep best, then SFT on curated data (lr=5e-5, 1 epoch)

This follows the approach from GRPO for Speech Recognition and Voxtral's own SFT→DPO training recipe.

Evaluation Results

Evaluated on 50 held-out test samples. Full benchmark (Evoxtral-Bench) with 7 metrics:

Core Metrics — Base vs SFT vs RL

SFT excels at raw transcription accuracy (best WER/CER). RL further improves expressive tag generation (+2.2% Tag F1, +3.3% Tag Recall, +2% Emphasis F1) at a small cost to WER.

Per-Tag F1 Breakdown (SFT → RL)

RL improved 9 tags, kept 4 stable, and regressed 3. Biggest gains on [clears throat] (+12.5%), [calm] (+100%), [sad] (+12.4%), and [shouts] (+25%).

Training Details

SFT Stage

RL Stage (RAFT)

Dataset

Custom synthetic dataset of 1,010 audio samples generated with ElevenLabs TTS v3:

• 808 train / 101 validation / 101 test
• Each sample has audio + tagged transcription with inline ElevenLabs v3 expressive tags
• Tags include: [sighs], [laughs], [whispers], [nervous], [frustrated], [clears throat], [pause], [excited], and more
• Audio encoder (Whisper-based) was frozen during training

Usage

import torch
from transformers import VoxtralForConditionalGeneration, AutoProcessor
from peft import PeftModel

repo_id = "mistralai/Voxtral-Mini-3B-2507"
# Use "YongkangZOU/evoxtral-lora" for SFT or "YongkangZOU/evoxtral-rl" for RL
adapter_id = "YongkangZOU/evoxtral-rl"

processor = AutoProcessor.from_pretrained(repo_id)
base_model = VoxtralForConditionalGeneration.from_pretrained(
    repo_id, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

# Transcribe audio with expressive tags
inputs = processor.apply_transcription_request(
    language="en",
    audio=["path/to/audio.wav"],
    format=["WAV"],
    model_id=repo_id,
    return_tensors="pt",
)
inputs = inputs.to(model.device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
transcription = processor.batch_decode(
    outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
print(transcription)
# [nervous] So... I was thinking maybe we could [clears throat] try that new restaurant downtown?

API

A serverless API with Swagger UI is available on Modal:

curl -X POST https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/transcribe \
    -F "[email protected]"

• Swagger UI
• Live Demo (HF Space)

W&B Tracking

All training and evaluation runs are tracked on Weights & Biases:

• SFT Training
• RL Training (RAFT)
• Base model eval
• SFT model eval
• RL model eval
• Project dashboard

Supported Tags

The model can produce any tag from the ElevenLabs v3 expressive tag set, including:

[laughs] [sighs] [gasps] [clears throat] [whispers] [sniffs] [pause] [nervous] [frustrated] [excited] [sad] [angry] [calm] [stammers] [yawns] and more.

Limitations

• Trained on synthetic (TTS-generated) audio, not natural speech recordings
• ~20% tag hallucination rate — model occasionally predicts tags not in the reference
• Rare/subtle tags ([calm], [confused], [scared]) have low accuracy due to limited training examples
• RL variant trades ~0.65% WER for better tag accuracy
• English only
• Best results on conversational and emotionally expressive speech

Citation

@misc{evoxtral2026,
  title={Evoxtral: Expressive Tagged Transcription with Voxtral},
  author={Yongkang Zou},
  year={2026},
  url={https://huggingface.co/YongkangZOU/evoxtral-lora}
}