mistral-smm4h2025-task6-vaem

Mistral VAEM Classifier (Task 6 - SMM4H-HeaRD 2025)

This model was developed by our team as part of our participation in the SMM4H-HeaRD 2025 Task 6. Task 6 focused on detecting vaccine adverse event mentions (VAEMs) in English-language Reddit posts related to herpes zoster (shingles) vaccination.

We fine-tuned the model using an iterative error-driven data augmentation strategy, in which misclassified examples were paraphrased and added back to the training data.

Our approach enabled the Mistral model to achieve strong performance, outperforming the RoBERTa baseline and achieving an F1 score of 0.96 on the official Task 6 test set.

📄 See our system description paper: https://workshop-proceedings.icwsm.org/abstract.php?id=2025_56

👩‍💻 Authors: Farnaz Zeidi, Roman Christof, Renate König, Liam Childs

---

🧠 Model Details

• Base model: mistralai/Mistral-Nemo-Instruct-2407
• Fine-tuning method: Supervised fine-tuning with iterative error-driven augmentation, where paraphrased false predictions were added back into the training data in multiple rounds
• Languages: English
• Task: Binary classification — determine whether a Reddit post personally reports an adverse reaction to a herpes zoster (shingles) vaccine (1) or not (0)
• Training data: Public SMM4H-HeaRD 2025 Task 6 dataset

---

📊 Evaluation

Performance on the SMM4H-HeaRD 2025 Task 6 test dataset:

---

💻 How to Use

You can use the model for inference by formatting the input using the same prompt template used during training.

📥 Download the Model from Hugging Face

from huggingface_hub import snapshot_download
from pathlib import Path

# Set the local download directory
mistral_models_path = Path.home().joinpath('mistral_models', 'smm4h2025-task6-vaem')
mistral_models_path.mkdir(parents=True, exist_ok=True)

# Download model files from Hugging Face
snapshot_download(
    repo_id="pei-germany/mistral-smm4h2025-task6-vaem",
    allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"],
    local_dir=mistral_models_path
)

🔧 Inference Example

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest

# Input text
user_text = "Got the shingles shot and had chills all night. Feeling awful."

# Prompt construction
prompt = f'''### Task
Determine whether the given text mentions an Adverse Drug Event (ADE) associated with a herpes zoster (shingles) vaccine.

### Definitions
- Adverse Drug Event (ADE): A physical or mental abnormality observed by a patient or clinician and suspected to be caused by a medication or vaccine.
- Zoster vaccine: A vaccine intended to prevent herpes zoster (shingles), caused by the reactivation of the varicella-zoster virus.

### Classification Guidelines
- The input is typically a social media post (e.g., tweet) authored by a patient or healthcare professional.
- Indirect mentions of vaccine use or ADEs (e.g., implied through symptoms or timing) should be considered.
- Classify as 1 (Positive) if:
  - An ADE is explicitly mentioned or strongly implied
  - There is a clear association with a zoster vaccine
- Classify as 0 (Negative) if:
  - There is no mention of a zoster vaccine, or
  - There is no mention (explicit or implied) of an ADE

### Output Format
Return only one of the following:
- 1 → ADE related to a zoster vaccine is mentioned
- 0 → No ADE or no link to a zoster vaccine

### Input:
{user_text}'''

# Load tokenizer and model
tokenizer = MistralTokenizer.from_file(mistral_models_path / "tekken.json")
model = Transformer.from_folder(str(mistral_models_path))

# Encode prompt and generate prediction
completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=1500, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)

# Decode and print result
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
print(result)