lettuce-thymos-26m-v1

lettuce-thymos-26m-v1

Multi-label emotion classification over the 28 GoEmotions labels in 22 languages. 26.5M parameters, 28 MB as int8 ONNX.

Thymos (θυμός) is the Homeric Greek word for the seat of emotion.

Model description

A transformer encoder distilled from FacebookAI/xlm-roberta-large. Label set and label order match SamLowe/roberta-base-go_emotions.

Intended uses and limitations

Emotion tagging of short conversational text in 22 languages, on device or on CPU. Multi-label: one input can carry several emotions.

Not for clinical, diagnostic or safety-critical use, and not a measure of a person's emotional state.

Known weaknesses are listed under Limitations.

How to use

import json, numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Zeolit/lettuce-thymos-26m-v1")
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])
meta = json.load(open("labels.json"))

enc = tok(["I'm so grateful you were there for me."], return_tensors="np")
probs = sess.run(None, {"input_ids": enc["input_ids"].astype(np.int64),
                        "attention_mask": enc["attention_mask"].astype(np.int64)})[0][0]

active = [(label, float(p))
          for label, p, t in zip(meta["labels"], probs, meta["thresholds"]) if p >= t]

Use the per-class thresholds in labels.json. A fixed 0.5 cutoff costs a large amount of macro-F1 because the rare classes rarely reach it.

Inputs longer than 128 tokens: split into overlapping windows with stride 96, then aggregate element-wise max for label presence, or length-weighted mean for overall tone. inference.py in this repository implements this; thresholds were fitted on single windows, so document-level aggregation is not calibrated.

Repository layout:

config.json               architecture config
model.safetensors         PyTorch weights
modeling_thymos.py        model definition, load with load_student(".")
inference.py              chunked inference helper
labels.json               28 labels, per-class thresholds, calibration record
tokenizer.json            XLM-RoBERTa SentencePiece, trimmed vocabulary
onnx/model.onnx           fp32 export
onnx/model_quantized.onnx int8 export
evaluation/               per-model evaluation artifacts for every table below

Training data

en, es, fr, de, pt, it, nl, ru, pl, uk, ar, fa, hi, bn, zh, ja, ko, vi, th, id, sw, tr.

Training procedure

Teacher. FacebookAI/xlm-roberta-large fine-tuned on all 22 languages. 3 epochs, batch 128, lr 1e-5 cosine with 6% warmup, layer-wise lr decay 0.9, EMA 0.999, max length 96, bf16.

Student. Distilled from teacher logits. 20 epochs, batch 512, lr 3e-4, AdamW weight decay 0.01. Loss: 0.7 per-label binary KD at temperature 2, plus 0.3 supervised. Embedding initialized from a low-rank SVD of the teacher's embedding matrix. Checkpoint selected on threshold-free validation macro-AP.

Thresholds. Per class, grid searched on held-out validation folds.

Quantization. The embedding table is quantized to int8 with per-row scales separately from the matmuls, since ONNX Runtime dynamic quantization does not cover Gather.

Evaluation

Five GoEmotions models sharing the same 28 labels, measured under one protocol. Rows ordered by parameter count.

GoEmotions

Test split of GoEmotions, machine-translated into 21 further languages, 119,394 rows. English thresholds tuned on English validation; multilingual thresholds tuned on the pooled validation set.

Teacher macro-F1 on the 22-language evaluation is 0.462.

Per language, this model against SamLowe/roberta-base-go_emotions, pooled thresholds:

BRIGHTER

BRIGHTER (SemEval-2025 Task 11): 28 languages, text written natively in each language, human annotated, CC-BY 4.0. No model in the table was trained on it.

The 28 GoEmotions labels are projected onto BRIGHTER's six emotions through Ekman groups, taking the maximum within each group. Thresholds tuned on each language's own dev split. Zero-shot across dataset, domain and label set.

Languages are grouped by whether they appear in this model's training set. Four of the unseen languages (Makhuwa, Kinyarwanda, Tatar, Nigerian Pidgin) are also absent from XLM-RoBERTa's pretraining corpus.

This model per language, training languages:

Unseen languages, highest to lowest: Romanian 0.516, Javanese 0.432, Marathi 0.414, Sundanese 0.397, Nigerian Pidgin 0.395, Tatar 0.310, Swedish 0.287, Hausa 0.283, Afrikaans 0.257, Igbo 0.243, Xhosa 0.240, Kinyarwanda 0.195, Yoruba 0.151, Zulu 0.134, Makhuwa 0.121.

Systems fine-tuned on BRIGHTER's training data score 0.55 to 0.89 on this task. Those numbers are not comparable to the zero-shot results above.

English macro-F1 on both evaluations:

Latency

ONNX Runtime, CPU, Intel i7-10700K, batch 1, median of 40 runs.

Chunked input scales close to linearly, 12 to 14 ms per 128-token window at 2 threads.

Limitations

grief, relief, pride, nervousness and embarrassment score well below the average. GoEmotions contains 77 grief training examples, and macro-F1 weights that class equally with classes that have thousands.

Arabic and Swahili trail the other languages on both evaluations. BRIGHTER tests Arabic in Algerian and Moroccan dialects while training used Modern Standard Arabic; the two dialects differ by 0.14.

Training domain is Reddit comments: short, informal, conversational. Literary text, long-form writing and formal registers are out of distribution.

Output probabilities are not calibrated as absolute confidences. Decisions come from the per-class thresholds.

Non-English training data is machine-translated. In the GoEmotions evaluation the test data comes from the same translator, so those per-language scores measure performance on translated text. Its 119,394 rows are 5,427 source comments rendered in 22 languages, not 119,394 independent examples. The BRIGHTER evaluation covers 11 of the 22 languages on natively written text.

Languages outside the listed 22 rely on backbone transfer and score lower.

License

Apache-2.0.

Upstream: GoEmotions (Apache-2.0), M2M-100 (MIT), XLM-RoBERTa (MIT). Evaluation data: GoEmotions (Apache-2.0), BRIGHTER (CC-BY 4.0).