laya-crisis-triage

laya-crisis-triage

Laya's multilingual checkpoint (mmBERT-base, 322M), fine-tuned to triage disaster messages: what people need, which kind of information a crisis tweet gives, and which humanitarian sector a report excerpt is about. One model answers all of these typed questions in one forward pass, in any of the languages above, with calibrated probabilities.

Research artefact from crisis-triage, where every number below comes from. It is not an emergency system: keep a human in the loop for any real decision about people in need.

Use

# pip install laya==0.3.5
import laya

agent = laya.load("mazenDDr/laya-crisis-triage")
questions = {
    "kind": {
        "type": "choice",
        "instructions": "Which kind of information does this crisis message give?",
        "criteria": {
            "requests or needs": "someone asks for help, food, water or supplies",
            "people affected": "people injured, killed, missing or affected",
            "infrastructure damage": "damaged buildings, roads, power or water supply",
            "displaced or evacuated": "people evacuated, displaced or in shelters",
            "donations and rescue work": "rescue work, volunteering, donations, aid response",
            "caution and advice": "warnings, advice, safety instructions",
            "sympathy and support": "prayers, sympathy, emotional support",
            "other or not humanitarian": "other news, or not about the disaster"
        }
    },
    "water_food": {"type": "noul", "instructions": "Does the sender ask for water food (drinking water or food)?"},
}
res = agent.predict({"message": "We have had no drinking water for two days in Leogane"}, questions)
print(res["answers"]["kind"]["probabilities"], res["answers"]["water_food"]["noul"])

It was trained on exactly these question wordings (and the other needs and sectors in questions.py). Other wordings still work, since it is Laya, but were not measured.

Results on held-out test data

Every system answers the same test messages. The splits are by time where there are several disasters (tuned on 2016-17 / 2011-12 events, tested on later ones). Macro-F1 is over 8 shared classes; mean AUC is over 4 needs (Haiti) or 10 sectors (HumSet). 95% intervals and paired differences are in results/t05_test.json.

Calibration error (ECE, lower is better):

Baselines: e5 + LR is multilingual-e5-base embeddings with logistic regression, trained on the same dev data, with its regularisation chosen by cross-validation on dev. Qwen3-4B-Instruct-2507 and Gemma-3-4B-it are zero-shot, asked the same questions and read out from next-token probabilities.

Training

• Base: convaiinnovations/laya, multilingual subfolder.
• Data: dev splits only, 40,737 (message, question) examples: Haiti 2010 SMS needs (original or English text), HumAID and CrisisBench tweet classes, HumSet sectors. A fixed dev sample was held out for the temperature fit and the decision thresholds; test data was never used.
• Recipe: Laya's own (RLCD policy gradient with a proper-scoring reward, plus soft cross-entropy), 2 epochs, 17 minutes on one RTX 5060 Ti, bf16.
• Held-out loss: yes/no 0.6095 → 0.1466, choice 2.0975 → 0.7771 (before → after, with the fitted temperature).

Limitations

• Urgency was not trained and drifted: on the Haiti SMS, urgency AUC is 0.527 for this model vs 0.573 for zero-shot Laya (against a stand-in label built from rescue and medical needs).
• The Haiti SMS set is one event with a message-level split, and HumSet's test shares projects with its training data; both favour trained models. HumAID and CrisisBench are the new-disaster tests.
• Only 11 Haiti test messages ask for rescue, so that need's score is noisy.
• Haitian Creole is read far better through a translation (see the repository).
• Speed, one message at a time on an RTX 5060 Ti under WSL: 16-53 ms (the GPU is bimodal between runs).

Licence

CC BY-NC-SA 4.0, because the model was trained on HumAID and CrisisBench (CC BY-NC-SA 4.0). The base model Laya is Apache-2.0 and HumSet is Apache-2.0. The Disaster Response Messages dataset (Appen) states no licence on its dataset card.