hallucination_detect

DeBERTa-v3-base Fine-Tuned for Hallucination Detection

Model Details Model Name: DeBERTa-v3-base Architecture: DeBERTa (Decoding-enhanced BERT with disentangled attention) Base Model: DeBERTa-v3-base Fine-tuned Dataset: PAWS (Paraphrase Adversaries from Word Scrambling) Task: Sentence Pair Classification (Hallucination Detection) Model Description This model is a fine-tuned version of the DeBERTa-v3-base model specifically for the task of detecting hallucinations between pairs of sentences. Hallucinations in this context refer to statements or information present in one sentence but not supported or contradicted by the other.

Fine-Tuning Dataset Dataset Name: PAWS (Paraphrase Adversaries from Word Scrambling) Dataset Description: The PAWS dataset contains pairs of sentences with high lexical overlap but different meanings, designed to challenge models' understanding of semantic content.

Dataset: https://huggingface.co/datasets/paws Training Procedure Number of Epochs: 10 Hardware Used: NVIDIA -A 100

Performance: Accuracy: 94.88% F1 Score: 92.3% Precision: 92.82% Recall: 95.81%

from transformers import AutoTokenizer, AutoModelForSequenceClassification import torch

Load model and tokenizer

tokenizer = AutoTokenizer.from_pretrained("Varun-Chowdary/hallucination_detect") model = AutoModelForSequenceClassification.from_pretrained("Varun-Chowdary/hallucination_detect")

Define the sentences

sentence1 = "Maradona was born in Argentina, South America." sentence2 = "Maradona was born in Brazil, South America. "

Tokenize and prepare input

inputs = tokenizer(sentence1, sentence2, return_tensors='pt', truncation=True, padding=True)

Perform inference

with torch.no_grad(): outputs = model(**inputs) logits = outputs.logits probabilities = torch.softmax(logits, dim=1)

Get the predicted label

predicted_label = torch.argmax(probabilities, dim=1).item() labels = ["No Hallucination", "Hallucination"] print(f"Predicted label: {labels[predicted_label]}")