MMedLM

MMedLM

💻Github Repo 🖨️arXiv Paper

The official model weights for "Towards Building Multilingual Language Model for Medicine".

MMedLM 2 has been released now. MMedLM2 is a more powerful multilingual medical foundation model, which has undergone the same medical data enhancement pipeline as MMedLM.

Introduction

This repo contains MMedLM, a multilingual medical foundation model with 7 billion parameters. MMedLM builds upon the foundation of InternLM and has been further pretrained on MMedC, a comprehensive multilingual medical corpus. This further pretraining enhances the model's medical-domain knowledge.

The model underwent further pretraining on MMedC with the following hyperparameters:

• Iterations: 15000
• Global batch size: 512
• Cutoff length: 2048
• Learning rate: 2e-5

The model can be loaded as follows:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMedLM", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Henrychur/MMedLM", torch_dtype=torch.float16, trust_remote_code=True)

• Note that this is a foundation model that has not undergone instruction fine-tuning.
• Testing has found that using the latest version of transformers will result in errors. It is recommended to use transformers==4.28.1.

News

[2024.2.21] Our pre-print paper is released ArXiv. Dive into our findings here.

[2024.2.20] We release MMedLM and MMedLM 2. With an auto-regressive continues training on MMedC, these models achieves superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.

[2023.2.20] We release MMedC, a multilingual medical corpus containing 25.5B tokens.

[2023.2.20] We release MMedBench, a new multilingual medical multi-choice question-answering benchmark with rationale. Check out the leaderboard here.

Evaluation on MMedBench

The further pretrained MMedLM 2 showcast it's great performance in medical domain across different language.

• GPT and Gemini is evluated under zero-shot setting through API
• Open-source models first undergo training on the trainset of MMedBench before evaluate.

Contact

If you have any question, please feel free to contact qiupengcheng@pjlab.org.cn.

Citation

@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}