deid_roberta_i2b2

Model Description

• A RoBERTa [Liu et al., 2019] model fine-tuned for de-identification of medical notes.
• Sequence Labeling (token classification): The model was trained to predict protected health information (PHI/PII) entities (spans). A list of protected health information categories is given by HIPAA.
• A token can either be classified as non-PHI or as one of the 11 PHI types. Token predictions are aggregated to spans by making use of BILOU tagging.
• The PHI labels that were used for training and other details can be found here: Annotation Guidelines
• More details on how to use this model, the format of data and other useful information is present in the GitHub repo: Robust DeID.

How to use

• A demo on how the model works (using model predictions to de-identify a medical note) is on this space: Medical-Note-Deidentification.
• Steps on how this model can be used to run a forward pass can be found here: Forward Pass
• In brief, the steps are:
  • Sentencize (the model aggregates the sentences back to the note level) and tokenize the dataset.
  • Use the predict function of this model to gather the predictions (i.e., predictions for each token).
  • Additionally, the model predictions can be used to remove PHI from the original note/text.

Dataset

• The I2B2 2014 [Stubbs and Uzuner, 2015] dataset was used to train this model.

Training procedure

• Steps on how this model was trained can be found here: Training. The "model_name_or_path" was set to: "roberta-large".

  • The dataset was sentencized with the en_core_sci_sm sentencizer from spacy.
  • The dataset was then tokenized with a custom tokenizer built on top of the en_core_sci_sm tokenizer from spacy.
  • For each sentence we added 32 tokens on the left (from previous sentences) and 32 tokens on the right (from the next sentences).
  • The added tokens are not used for learning - i.e, the loss is not computed on these tokens - they are used as additional context.
  • Each sequence contained a maximum of 128 tokens (including the 32 tokens added on). Longer sequences were split.
  • The sentencized and tokenized dataset with the token level labels based on the BILOU notation was used to train the model.
  • The model is fine-tuned from a pre-trained RoBERTa model.

• Training details:

  • Input sequence length: 128
  • Batch size: 32 (16 with 2 gradient accumulation steps)
  • Optimizer: AdamW
  • Learning rate: 5e-5
  • Dropout: 0.1

Results

Questions?

Post a Github issue on the repo: Robust DeID.