HealthHistoryOpenCabrita3B-pt-ft

Health History OpenCabrita3B-pt-ft

The HealthHistoryOpenCabrita3B-pt-ft was fine-tuned on the pre-trained Portuguese model open-cabrita3b and with patient data from health insurances organized in the form of historical sentences. The initial objective of the training was to predict hospitalizations, however, due to the possibility of applications in other tasks, we made these models available to the scientific community. This model was trained in Portuguese. There are also other training approaches that can be seen at:

Other pre-trained models

• HealthHistoryRoBERTa-en
• HealthHistoryRoBERTa-pt
• HealthHistoryBERT-en
• HealthHistoryBioBERT-en
• HealthHistoryBio_ClinicalBERT-en
• HealthHistoryBERTimbau-pt

Other Models trained to predict hospitalizations (fine-tune)

• HealthHistoryOpenLLaMA3Bv2-en-ft
• HealthHistoryOpenLLaMA7Bv2-en-ft
• HealthHistoryOpenLLaMA13B-en-ft
• HealthHistoryOpenCabrita3B-pt-ft
• HealthHistoryRoBERTa-en-ft
• HealthHistoryRoBERTa-pt-ft
• HealthHistoryBERTimbau-pt-ft
• HealthHistoryBERT-en-ft
• HealthHistoryBioBERT-en-ft
• HealthHistoryBio_ClinicalBERT-en-ft

Fine-tune Data

The model was fine-tuned from 83,715 historical sentences from health insurance patients generated using the approach described in this paper Predicting Hospitalization from Health Insurance Data.

Model Fine-tune

Fine-tune Procedures

The model was fine-tuned on a GeForce NVIDIA RTX A5000 24GB GPU from laboratories of IT departament at UFPR (Federal University of Paraná). Fine-tuning was done using LoRA (Low Rank Adaptation), a parameter-efficient fine-tuning (PEFT) that minimizes computational and memory overhead, making deployments in resource-constrained scenarios more practical, while preserving the knowledge of the pre-trained model.

Fine-tune Hyperparameters

We use a batch size of 4, a maximum sequence length of 2048 tokens, accumulation steps of 16, masked language model probability = 0.15, number of epochs = 1 and a learning rate of 5.10−5 to fine-tune this model.

For the Lora hyperparameters use lora_r = 8, lora_alpha=32 and lora_dropout=0.1.

Fine-tune time

The training time was 45 hours 5 minutes and 28 seconds per epoch.

Time to predict

Time to predict the first 500 sentences of dataset data_test_seed_pt_12.csv: 62.97 seconds

Time to predict the first 500 sentences + data tokenization of data_test_seed_pt_12.csv: 198.91 seconds

Predictions made with the maximum sentence length allowed by the models.

How to use the model

Load the model via the transformers library:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("efbaro/HealthHistoryOpenCabrita3B-pt-ft")
model = AutoModel.from_pretrained("efbaro/HealthHistoryOpenCabrita3B-pt-ft")

After loading, some settings need to be adjusted to use the model correctly. This is because the special token ['PAD'] was included during training, which changed the size of the final model's vocabulary. To ensure the model works properly, it is necessary to resize the token embeddings according to the new vocabulary size. This can be done with the following code:

model.resize_token_embeddings(len(tokenizer))

It is expected that sentence truncation will always be performed on the left side, ensuring that the most recent data is preserved in the historical sentence. This can be configured with the following code:

tokenizer.truncation_side = 'left';

Below is the code for loading and configuration:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("efbaro/HealthHistoryOpenCabrita3B-pt-ft")
tokenizer.truncation_side = 'left';

model = AutoModel.from_pretrained("efbaro/HealthHistoryOpenCabrita3B-pt-ft")
model.resize_token_embeddings(len(tokenizer))

Load the model with quantization

When you need to use the model in a memory-constrained environment, load it with 4-bit quantization using the bitsandbytes library.

from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig
import torch

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("efbaro/HealthHistoryOpenCabrita3B-pt-ft")
tokenizer.truncation_side = 'left';

# 4-bit quantization configuration
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # Load the model in 4-bit
    bnb_4bit_use_double_quant=True,  # Apply double quantization for better efficiency
    bnb_4bit_quant_type="nf4",  # Quantization type (nf4 is recommended)
    bnb_4bit_compute_dtype=torch.bfloat16  # Data type for computation (bfloat16 is efficient)
)

# Load the model with quantization
model = AutoModel.from_pretrained(
    "efbaro/HealthHistoryOpenCabrita3B-pt-ft",
    quantization_config=bnb_config
)
model.resize_token_embeddings(len(tokenizer))

More Information

Refer to the original paper, Predicting Hospitalization with LLMs from Health Insurance Data

Refert to another article related to this research, Predicting Hospitalization from Health Insurance Data

Questions?

Email:

• Everton F. Baro: [email protected], [email protected]
• Luiz S. Oliveira: [email protected]
• Alceu de Souza Britto Junior: [email protected]