stigmatizing_detector

Overview

This model can be used to evaluate how well doctors are adhereing to proper clinical documentation practices. It specifically analyzes their text for uses of stigmatizing language.

Background

When documenting their patient's conditions, doctors are trained to eliminate the use of stigmatizing language, which are negative labels that can cause shame and discrimination against individuals with certain conditions. Countless advocacy groups like the National Institute on Drug Abuse and the American Diabetes Association have posted word banks of words to avoid and recommended alternatives (for example, "drug abuser" should be replaced by "person who uses drugs"). However, these word banks do not cover the breadth of the human language.

Research Problem

Researchers have developed machine learning detectors, but the training of these detectors requires thousands of hand-annotated clinical notes, a costly and labor-intensive process. This model uses a novel framework that only requires 300 example sentences of stigmatizing and non-stigmatizing language to train.

Model Training

To generate the stigmatizing and non-stigmatizing training sentences, I promped ChatGPT to generate 300 stigmatizing and non-stigmatizing sentences using the word banks published by the American Drug Association and the National Institute on Drug Abuse. However, this is far from enough to fine-tune a large language model. So I fine-tuned a BERT large language model on a dataset of Google Play app reviews. Using this fine-tuned model, I performed sentiment analysis on the clinical sentences, and performed classification on the sentence embeddings and sentiment value.

Model Infrastructure

For each sentence that gets parsed, we feed it into the default tokenizer for BERT, and send the tokens into the fine-tuned BERT large language model, which returns a 1 for positive sentiment or 0 for negative. To retrieve the sentence embeddings, we found that summing all embeddings for each non-padding token performed the best as our "sentence embedding." We then take the embedding with dimension 768, which is the default used by BERT, and performed PCA dimensionality reduction to reduce the dimensions to 50. Appending the sentiment value onto the end of this vector, we now have a vector of dimension 51 which we can perform logistic regression on to calculate and output the probability the sentence is considered "stigmatizing."