NLP-A3

MedSoLLM: Medical Socratic Dialogue Large Language Model

Project Overview

MedSoLLM is a large language model specifically designed for the medical domain that employs a Socratic dialogue approach. It aims to guide patients through questions to provide more detailed and specific symptom information while avoiding making explicit medical diagnoses and treatment recommendations. The model follows the "Socratic questioning" method, using open-ended questions to guide patients in self-exploration and expression, collecting more comprehensive symptom information.

Core Features

• Socratic Questioning: Skillfully guides patients to provide richer, more detailed symptom descriptions
• Medical Safety: Strictly avoids making medical diagnoses and treatment recommendations
• Empathetic Communication: Maintains professional, respectful, and warm communication style
• Based on Llama-3-8B-UltraMedical: Uses TsinghuaC3I/Llama-3-8B-UltraMedical as the base model
• KTO Training Method: Employs KTO (Kahneman-Tversky Optimization) training method to optimize medical dialogue capabilities

Project Structure

MedSoLLM/
├── dataset/                  # Training datasets
│   ├── 6867.json            # Negative samples
│   └── 7517.json            # Positive samples
├── train/                    # Training-related code
│   ├── Llama-3-8B-UltraMedical-KTO-LoRA/  # Trained model output
│   ├── KTO_train_custom.py   # KTO training core code
│   ├── start_kto_training.sh # Training startup script
│   ├── KTO.py                # KTO implementation
│   ├── check_kto_params.py   # KTO parameter check
│   └── test_kto_config.py    # KTO configuration test
├── evaluation_results/       # Evaluation results
│   ├── evaluation_summary.json    # Evaluation summary
│   └── sample_*_eval.json        # Detailed evaluation results for individual samples
├── random_samples/           # Random dialogue samples
│   └── sample_*.json         # Dialogue sample files
├── logs/                     # Training and evaluation logs
├── scripts/                  # Auxiliary scripts
├── evaluate.py               # Model evaluation script
├── eval_readme.md            # Evaluation dimensions and criteria explanation
└── README.md                 # Project documentation

Evaluation Dimensions

The model is evaluated across six dimensions, each using a 1-5 scale:

1. Dialogue Quality: Assesses whether the model can guide patients to provide more specific and detailed symptom information through questioning
2. Overall Effectiveness: Evaluating the overall quality of the medical consultation dialogue.

Training Method

This project uses the KTO (Kahneman-Tversky Optimization) training method, which is a preference-based training approach that enhances model capabilities through contrastive learning with positive and negative samples.

Training Parameters:

• Base Model: TsinghuaC3I/Llama-3-8B-UltraMedical
• Quantization Method: 4-bit quantization
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
  • r=16
  • lora_alpha=32
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training Epochs: 3
• Batch Size: 12
• Learning Rate: 2e-5

Evaluation Results

Based on the evaluation of 100 dialogue samples, the model achieved the following average scores:

• Dialogue Quality: 3.96/5.0
• Reasoning Chain Quality: 3.74/5.0
• Premature Diagnosis Avoidance: 3.93/5.0
• Overall Score: 3.88/5.0

Usage Instructions

Training the Model

1. Clone this repository
2. Prepare positive and negative sample datasets
3. Run the training script:

cd train
chmod +x start_kto_training.sh
./start_kto_training.sh

Evaluating the Model

Use the evaluate.py script to evaluate the model:

python evaluate.py --model_path "Llama-3-8B-SFT-Med" --sample_dir "random_samples" --output_dir "evaluation_results"

Example Dialogue

Patient: "Doctor, I was diagnosed with multiple bilateral breast cystic hyperplasia at a local hospital. They did an ultrasound, but I'm still worried. Could you explain what this means?"

Assistant: "I understand your concern. Let's look at the results together. The color Doppler ultrasound shows scattered low-echo bands forming nodular structures and multiple cysts. The largest cyst on the left is 25×5mm. Importantly, no blood flow was detected in these cysts. Do you think this lack of blood flow might indicate what?"

Patient: "Does that mean they're not cancerous? I heard that tumors usually have blood flow."

Assistant: "You're right. The lack of blood flow is a reassuring sign. The elasticity score of 3 also indicates benign characteristics. The mammogram shows dense tissue and some calcification, which is common in hyperplasia. The 1 cm nodule on the right breast looks well-defined—that's also a good sign. Based on these findings, do you think which treatment method would be most appropriate?"

Patient: "I'm not sure... Surgery looks scary, but should we remove them?"

Assistant: "That's a good question. Since the cysts are small (all less than 3 cm) and asymptomatic, and considering that ultrasound is more accurate than mammography for cysts, I recommend conservative management and regular monitoring. We'll do follow-up ultrasound to check for changes. Only when you experience pain or the cysts grow significantly will we consider surgery. Does this approach make sense to you?"

License

[Here, please enter the license information]

Contribution

Welcome to improve the project by submitting Pull Requests or Issues.

Acknowledgment

Thank you to TsinghuaC3I for providing the Llama-3-8B-UltraMedical base model, and all team members who contributed to this project.