First_Aid_Assistant_Evaluation

🏥 First Aid Tutor

📘 About the Project

First Aid Tutor is an AI-powered chatbot designed to evaluate the efficacy of Retrieval-Augmented Generation (RAG) in medical education for paramedical and medical students. It provides first-aid guidance based on medical literature to assess learning outcomes and response accuracy.

This repository contains the codebase for the First Aid Assistant chatbot, tested on 10 first-aid-related questions.

📢 Ownership and Attribution

⚠️ Disclaimer:

• The First Aid Assistant chatbot is the property of Mysore Medical College and Research Institute.
• The uploaded FIRST AID PDF that forms the basis of the Retrieval-Augmented Generation (RAG) process is an official course material and the property of Mysore Medical College and Research Institute.

---

🚑 First Aid Tutor - RAG Chatbot

The First Aid Tutor is a Retrieval-Augmented Generation (RAG) chatbot designed to assist medical and paramedical students in learning first-aid protocols. It is strictly trained on a verified first-aid guide and only provides information present in the First Aid PDF.

🎯 Features

✅ Evidence-Based Responses - The chatbot only retrieves and generates answers based on the First Aid PDF.
✅ User-Friendly Interface - Developed using Gradio, featuring a modern and intuitive UI.
✅ Medical Education Research - Used for evaluating RAG efficacy in medical education.

🔬 How It Works

• Uses TF-IDF + Cosine Similarity for retrieving contextually relevant information.
• Queries are answered based strictly on verified medical content.
• Integrated with Gradio UI for an easy-to-use chatbot experience.

📌 Live Demo

🚀 Try the First Aid Tutor Chatbot

📥 How to Run Locally

You can run the chatbot locally using the following commands:

git clone https://github.com/SYEDFAIZAN1987/First-Aid-Tutor
cd First-Aid-Tutor
pip install -r requirements.txt
python raggpt.py

🩺 First Aid Assistant - Model Evaluation Report

This repository presents the evaluation results of the First Aid Assistant chatbot, which provides first aid guidance based on common emergency conditions. The model has been evaluated using the RAGAS framework with metrics that assess the quality of the generated answers.

📊 Evaluation Metrics

The chatbot was evaluated based on the following RAGAS metrics:

• Answer Relevancy: Measures how relevant the response is to the user's question.
• Answer Correctness: Compares the generated response to the ground truth to assess factual correctness.
• Semantic Similarity: Evaluates how semantically similar the generated answer is to the reference answer.

---

🚀 Performance Summary

---

📈 Detailed Results

Here’s a snapshot of the evaluation for some sample questions:

---

📋 Key Insights

• The chatbot performed exceptionally well in semantic similarity (average score of 0.97), indicating that responses are closely aligned with the ground truth.
• Answer correctness is strong overall but showed slight variability, suggesting room for improvement in handling complex queries.
• The relevancy of responses remained consistently high, reflecting the model's ability to address user questions effectively.

---

📝 Evaluation Artifacts

• RAGAS Evaluation Report: View Full Report

---

🌟 Conclusion

The First Aid Assistant demonstrates reliable performance in answering first aid-related queries with high semantic accuracy and relevancy. Continuous improvement in factual correctness will further enhance its capability to provide life-saving information in emergency situations.

---

📥 Download & Re-Evaluate

You can re-evaluate the chatbot by running the following:

git clone https://huggingface.co/DrSyedFaizan/First_Aid_Assistant_Evaluation
cd First_Aid_Assistant_Evaluation
python eval.py

---

📌 Author

👨‍⚕ Dr. Syed Faizan | 🏥 Healthcare AI & Data Science
🔗 Hugging Face Repo: First_Aid_Assistant_Evaluation

📩 For collaborations, inquiries, or improvements, feel free to contribute or reach out!