mistral-smm4h2025-task1-ade

Mistral ADE Classifier (Task 1 - SMM4H-HeaRD 2025)

This model was developed by our team as part of our participation in the SMM4H-HeaRD 2025 Task 1. Task 1 focused on detecting adverse drug events (ADEs) in multilingual social media texts (German, French, Russian, and English).

We fine-tuned the model using an iterative error-driven data augmentation strategy, in which misclassified examples were paraphrased and added back to the training data.

Our approach enabled the Mistral model to outperform the RoBERTa baseline, achieving the highest macro-averaged F1 score (0.709) on the Task 1 test set—ranking first among all participating systems.

📄 See our system description paper: https://workshop-proceedings.icwsm.org/abstract.php?id=2025_56

👩‍💻 Authors: Farnaz Zeidi, Roman Christof, Renate König, Liam Childs

---

🧠 Model Details

• Base model: mistralai/Mistral-Nemo-Instruct-2407
• Fine-tuning method: Supervised fine-tuning with iterative error-driven augmentation, where paraphrased false predictions were added back into the training data in multiple rounds
• Languages: English, German, French, Russian
• Task: Binary classification — detect whether a post mentions an ADE (1) or not (0)
• Training data: Public SMM4H-HeaRD 2025 Task 1 dataset

---

📊 Evaluation

Performance on the SMM4H-HeaRD 2025 Task 1 test dataset:

---

💻 How to Use

You can use the model for inference by formatting the input using the same prompt template used during training:

📥 Download the Model from Hugging Face

from huggingface_hub import snapshot_download
from pathlib import Path

# Set the local download directory
mistral_models_path = Path.home().joinpath('mistral_models', 'smm4h2025-task1-ade')
mistral_models_path.mkdir(parents=True, exist_ok=True)

# Download model files from Hugging Face
snapshot_download(
    repo_id="pei-germany/mistral-smm4h2025-task1-ade",
    allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"],
    local_dir=mistral_models_path
)

🔧 Inference Example

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest

# Input text
user_text = "Nach der Einnahme von Ibuprofen habe ich starke Kopfschmerzen bekommen."

# Prompt construction
prompt = f'''### Task:
Classify whether the given text mentions an "Adverse Drug Event".

### Definition:
An "Adverse Drug Event" is an observation made by "patients or clinicians" about "physical or mental abnormalities" suspected to be caused by "medications or vaccines".

### Guidelines:
- The text is a tweet/post where a doctor or patient discusses a medical condition.
- Patients may indirectly imply they are using a medication that caused an ADE.
- If an ADE is explicitly or implicitly mentioned, classify it as 1 (Positive).
- If there is no mention of an ADE, classify it as 0 (Negative).

### Output Format:
Return only:  
1 → ADE mentioned  
0 → No ADE mentioned  

### Input:
{user_text}'''

# Load tokenizer and model
tokenizer = MistralTokenizer.from_file(mistral_models_path / "tekken.json")
model = Transformer.from_folder(str(mistral_models_path))

# Encode prompt and generate prediction
completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=1500, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)

# Decode and print result
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
print(result)