lncrna-biocontext-longformer

lncrna-biocontext

This model is designed to determine whether a given abstract talks about an lncRNA in the context of disease or not.

The model has been trained on data from lncBook-Wiki about papers which have been curated by experts based on the biological context they discuss. We have collected the abstracts for these papers and simplified the classification into disease/not disease. We then fine-tune a longformer model to make a binary classification.

We achieve pretty good results:

Though the test set is only 59 examples, with 22 discussing disease.

Key stats

The model size is ~600MB. It will run really fast on the MPS device of an M-series mac. It should also run pretty fast on a normal CPU.

The context windiw is 4,096 tokens which comes from the base longformer model. In training we limit the context to 1280 tokens because that was a bit bigger than the longest abstract we saw. Long abstracts may cause trouble.

Limitations

The base model is trained on MLM on wikipedia text (and maybe some other stuff). As such, it might not have a great understanding of scientific literature.

The dataset used to train this model was tiny at only 588 examples overall. This means only 470 samples for training with 59 for validation and testing. These have been deliberately sampled to be roughly equally distributed between the two classes.

The dataset these are derived from is also massively imbalanced, having 19,229 examples but only 294 that are not disease. As a result the model is trained on a dataset that hugely undersamples the disease context abstracts.

While the model has been tested on some abstracts derived from lncBook's annotations, it hasn't really been tested on 'wild' abstracts.

Next steps

The next step will be to be able to classify both the specific disease (e.g. lung adenocarcinoma), and the non-disease context (e.g. localisation) a paper discusses.