XLMRHallucinationDetectorSHROOMCAP

SVNIT-AGI at SHROOM-CAP 2025: Multilingual Hallucination Detection Model

Model Description

The model is an XLM-RoBERTa-Large based fine-tuned model for scientific hallucination detection across 9 languages using the Huggingface transformers library.

• Developed by: Harsh Rathva, Pruthwik Mishra, Shrikant Malviya
• Funded by: Sardar Vallabhbhai National Institute of Technology, Surat
• License: MIT
• Finetuned from model: xlm-roberta-large
• Competition: SHROOM-CAP 2025 Shared Task (2nd place in Gujarati)

Model Sources

• Repository: https://github.com/ezylopx5/SHROOM-CAP2025
• Paper: https://arxiv.org/abs/2511.18301

Uses

The model can be directly used for detecting hallucinations in scientific text across 9 languages:

• Training Languages: English (en), Spanish (es), French (fr), Hindi (hi), Italian (it)
• Zero-shot Languages: Bengali (bn), Gujarati (gu), Malayalam (ml), Telugu (te)

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "Haxxsh/XLMRHallucinationDetectorSHROOMCAP"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

def detect_hallucination(text):
    """Detect if text contains scientific hallucinations."""
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
    
    with torch.no_grad():
        outputs = model(**inputs)
        predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    
    label = "HALLUCINATED" if predictions[0][1] > 0.5 else "CORRECT"
    confidence = predictions[0][1].item() if label == "HALLUCINATED" else predictions[0][0].item()
    
    return {"label": label, "confidence": confidence}

# Example usage
test_texts = [
    "The protein folding mechanism involves quantum tunneling effects at room temperature.",
    "Water boils at 100°C at standard atmospheric pressure.",
    "Einstein discovered the theory of relativity in 1905 with his paper on special relativity."
]

for text in test_texts:
    result = detect_hallucination(text)
    print(f"Text: {text}")
    print(f"Prediction: {result['label']} (confidence: {result['confidence']:.4f})\n")

Label Mapping

• 0: CORRECT (factually accurate scientific text)
• 1: HALLUCINATED (contains factual errors or fabrications)

Downstream Use

Can be integrated into:

• Scientific writing assistants
• LLM output verification systems
• Academic paper review tools
• Multilingual fact-checking pipelines

Out-of-Scope Use

• The model is specifically trained for scientific domain text
• May not perform well on general domain hallucinations
• Limited to the 9 languages mentioned above

Limitations

• Performance varies across languages (best in Gujarati, competitive in others)
• Trained primarily on scientific text, may not generalize to other domains
• Requires domain adaptation for highly specialized scientific fields

Training Details

Training Data

• Total Samples: 124,821 balanced samples (50% correct, 50% hallucinated)
• Sources: Unified dataset from SHROOM-CAP, hallucination_dataset_100k, LibreEval, FactCHD
• Languages: 9 languages with cross-lingual transfer

Training Procedure

• Base Model: XLM-RoBERTa-Large (560M parameters)
• Training Regime: Full fine-tuning (not LoRA/PEFT)
• Training Batch Size: 32 with gradient accumulation
• Learning Rate: 2e-5
• Weight Decay: 0.01
• Epochs: 3
• Sequence Length: 256 tokens

Training Hyperparameters

{
    "per_device_train_batch_size": 16,
    "gradient_accumulation_steps": 2,
    "learning_rate": 2e-5,
    "num_train_epochs": 3,
    "max_seq_length": 256,
    "warmup_ratio": 0.1,
    "weight_decay": 0.01
}

Evaluation

Competition Results (SHROOM-CAP 2025)

Metrics

• Primary: Macro F1 Score
• Validation Performance: 0.8510 F1
• Competition Performance: ~0.40-0.51 F1 (due to distribution shift)

Compute Infrastructure

• Hardware: NVIDIA H200 GPU (141GB VRAM)
• Training Time: 1 hour 14 minutes
• Framework: PyTorch, HuggingFace Transformers

Model Size

• Parameters: 560M
• File format: SafeTensors
• Tensor type: F32

Acknowledgements

• SHROOM-CAP 2025 Organizers for the shared task
• Lightning AI for H200 GPU infrastructure
• HuggingFace for the XLM-RoBERTa-Large model
• All dataset contributors

---