humaid_distilbert_best

HumAID Disaster Tweet Classifier

Model Description

This model is a fine-tuned DistilBERT-base-uncased transformer trained to classify crisis and disaster-related tweets into 11 distinct humanitarian categoriesspan_0span_0. It was developed as the text-processing branch of a multimodal disaster response API pipelinespan_1span_1.

Dataset

The model was trained on the QCRI/HumAID-all datasetspan_2span_2.

• Scope: 76,484 labeled tweets covering 19 disaster events from 2016 to 2019span_3span_3.
• Language: Effectively English-only (non-ASCII characters represent only 0.3% of the dataset)span_4span_4.
• Splits: We utilized the 53,531 rows from the HumAID "train" split and re-split it 80/10/10 (42,824 train / 5,353 validation / 5,354 test) rather than using the predefined validation/test splitsspan_5span_5. Zero exact-duplicate leakage exists across these splitsspan_6span_6.

Performance Metrics

The model achieved the following on the test set:

• Accuracy: 75%span_7span_7.
• Macro F1: 0.73span_8span_8.
• Despite a severe 59.6x class imbalance, class weighting allowed the extreme minority class (missing_or_found_people, 25 test examples) to achieve a 0.72 F1 score and 0.88 recallspan_9span_9.

Known Limitations and Biases

• Vocabulary Overfit Risk: Top TF-IDF words per class are heavily dominated by specific event names (e.g., "irma", "maria", "kerala") rather than generalized humanitarian languagespan_10span_10. Because the data could not be split by event/date, the model may partly learn specific event vocabulary, which could limit its ability to generalize to future unseen eventsspan_11span_11.
• Semantic Overlap: The model showed weaker performance on the other_relevant_information class (0.49 F1) due to it being an inherently fuzzy catch-all categoryspan_12span_12. The requests_or_urgent_needs class (0.43 precision) is also likely confused with semantically adjacent rescue or sympathy classesspan_13span_13.