my-project-demo

Project: Digital Health Twin Prototype

Developed by: Nikhilesh Patil(Hugging Face: nikhilesh123)

This project is part of an initiative to explore the AI/ML healthcare sector.

AI/ML Digital Health Twin Prototype Application for Analyzing Medical Reports (PDF)

This application uses a Large Language Model (LLM) to analyze medical reports and images and generate text-based insights.

Digital Health Twin Prototype

A comprehensive AI-powered application that digitizes medical lab reports. This tool ingests raw medical PDFs, extracts structured data, provides AI-generated health insights, and generates a portable QR code for data interoperability.

Important Disclaimer

Please Read Before Use:

This application is a Proof of Concept (PoC) and an experimental prototype.

Accuracy: The PDF ingestion and OCR (Optical Character Recognition) mechanism may occasionally misinterpret data or structure. Always verify the extracted values against the original PDF.

Not Medical Advice: The summaries, precautions, and risk scores generated by the AI are for demonstration purposes only. They should not be used as a substitute for professional medical advice, diagnosis, or treatment.

Key Features:

• PDF Ingestion: Upload any standard medical lab report (PDF). The system analyzes the document structure to extract key health metrics (e.g., Glucose levels).

• AI Analysis: Uses advanced LLMs to generate: Easy-to-understand summaries of complex medical jargon.

Personalized precautions and lifestyle recommendations.

Routine follow-up suggestions.

• Structured Data: Converts unstructured PDF text into clean JSON format for easy integration.

• QR Code & Simulation: Generates a QR code containing the digitized health record. Includes a "Simulate QR Scan" feature to demonstrate how data can be retrieved instantly by healthcare providers.

Current Limitations & Future Roadmap

This project is a Proof of Concept (PoC) designed to demonstrate the feasibility of the architecture. Users should be aware of the following:

• OCR & Ingestion: The current extraction pipeline (Tesseract/Gemini) may struggle with:

Complex, multi-column table layouts in PDF reports.

Low-resolution or handwritten scans.

• LLM Hallucinations: Occasionally, the AI summary may misinterpret misread numerical values from the OCR step ("Garbage In, Garbage Out").

• Accuracy: The current extraction accuracy is estimated at ~60-70%.

Future Improvements:

• Implement RAG (Retrieval-Augmented Generation) to ground the AI responses in verifiable medical text.

• Fine-tune the vision model on a specific dataset of lab reports to improve table recognition.

• Add a "Human-in-the-Loop" validation step where users can correct the extracted JSON before generation.

How It Works

1. Upload: Drag and drop your medical lab report (PDF) into the application.
2. Analyze: The system processes the file, extracting values (e.g., "Glucose: 120 mg/dL") and calculating risk scores (e.g., "Diabetes Risk").
3. Review: Read the AI-generated summary and extracted JSON data.
4. Connect: A QR code is generated. Click "Simulate QR Scan" to see how this data is retrieved and displayed in a structured format.

Tech Stack -

User Interface: Streamlit For building the interactive web dashboard.

OCR & Text Extraction:

• pytesseract (Tesseract OCR) - For extracting text from scanned images/PDFs.

• pdf2image - Converts PDF pages into images for OCR processing.

• PyPDF2 - For parsing and analyzing PDF file structures.

• Image Processing: Pillow (PIL) - Handles image manipulation and analysis.

• Utilities: qrcode - Generates the scannable QR code for the digital health twin.

• LLM: For secure and private local inference, I employed the Meta Llama 3 8B Instruct LLM, quantized with Q4_K_M, running entirely on-device via llama.cpp.

• Programming Language: Python

How to Run the Application

To run this application locally, you must first ensure all dependencies are installed and the model file is in the correct location.

1. Install the core libraries:

   All required libraries are listed in the requirements.txt file. You can install them by running:

   pip install -r requirements.txt

2. Install the Llama library:

   The llama-cpp-python library needs to be installed separately to work with the .gguf model file.

   pip install llama-cpp-python

3. Model File Location:

   Make sure the model file, Meta-Llama-3-8B-Instruct-Q4_K_M.gguf, is located inside the application folder.

4. Run the Application:

   Once all dependencies are installed, you can start the Streamlit application with the following command: streamlit run app.py

Tips for Optimal Performance

For the best results, it is important to use clear and high-quality images. Blurred or low-quality images may affect the model's ability to accurately generate a response.

Here are examples of the kind of clear images that lead to optimal results:

Link to clear image example 1: https://drive.google.com/file/d/1CA5JtqushdmdsjRnsR4yHLNgEFzSpQqV/view?usp=sharing

Link to clear image example 2: https://drive.google.com/file/d/1aWMFusnw3R_kJv9eljvHzcwhzUQR40QP/view?usp=sharing

Link to clear image example 3: https://drive.google.com/file/d/1iL3Gt1rE0ocznFxxTEwx85dKUqDAUxhT/view?usp=sharing

⚖️ Architectural Comparison: Cloud vs. Edge

To address the critical trade-off between model capability and data privacy in healthcare, I engineered two distinct versions of this application.

Note on Accuracy: Both versions are experimental prototypes. While the Cloud version generally outperforms the Local version, both currently face challenges with low-resolution PDF ingestion.

1. Privacy-First Version (Current Repository)

• Architecture: Streamlit + Meta Llama 3 8B (Quantized) + llama.cpp.
• Focus: Prioritizes Zero-Trust Privacy. This version runs entirely offline on consumer hardware, ensuring patient data never leaves the device.
• Trade-off: Running an 8B parameter model locally forces a significant trade-off in reasoning capability compared to massive cloud models, resulting in more frequent hallucinations on complex data.
• Status: Source code and model implementation available here.

2. Cloud-Native Version (Alternative)

• Architecture: Gradio + Google Gemini (Multimodal).
• Focus: Leverages the Gemini 2.5 Flash model to analyze structured lab data. It transforms raw extracted values into clear, human-readable medical summaries and provides actionable next steps for patients.
• Status: Deployed as a live demo.
• [Link to Hugging Face Space] : https://huggingface.co/spaces/nikhilesh123/medical