FineTuned_RNAModels_for_BranchPoint_Prediction

Fine-Tuning RNA Language Models to Predict Branch Points

This repository contains several fine-tuned RNA language models for predicting branch points within intronic sequences. The models are fine-tuned using the MultiMolecule library and evaluated on experimental datasets.

The following RNA language models were fine-tuned:

• SpliceBERT
• RNABERT
• RNA-FM
• RNA-MSM
• ERNIE-RNA
• UTR-LM

The dataset contains 177980 samples and is an experimental-data only subset of the dataset used to train BPHunter.

It has been split into approximately 80/10/10 train/validation/test by chromosome type:

• Train: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr19, chr20, chr21, chr22, chrX, chrY,
• Validation: chr9, chr10
• Test: chr8, chr11

Training Details

Each model was trained on the full dataset for 3 epochs with a batch size of 16, except for RNA-FM, which required a reduced batch size of 12 due to VRAM limitations. The following hyperparameters were used for most models, including RNABERT, RNA-FM, RNA-MSM, and UTR-LM:

• Optimizer: AdamW
• Learning rate: 3e-4
• Weight decay: 0.001

However, SpliceBERT and ERNIE-RNA failed to converge using these parameters. To address this, we adjusted the hyperparmeters to:

• Learning rate: 2e-5
• Weight decay of 0.01

The adjustments were made based on empirical observations during early training. While ideally, comprehensive hyperparameter tuning would be done for each model to optimize perforance, this was not feasible within the scope of the project due to the high computational cost and training time required.

GitHub

All code used to create and evaluate this model can be found at this link.