RIBOSPAN-FM

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Many full-length RNAs, particularly mRNAs, exceed the ~1K context lengths used to pretrain representative dense RNA encoders, forcing long transcripts to be truncated and preventing their 5′ UTR, CDS, and 3′ UTR from being modeled jointly at single-nucleotide resolution. RiboSpan is a 1.61B-parameter bidirectional Transformer with single-nucleotide tokenization, dense self-attention in every layer, and native long-context pretraining at up to 10,240 nt. It is trained on 67.6 million RNA sequences comprising 85.7 billion nucleotides from RNAcentral, Ensembl, and Ensembl Genomes.

✨ Features

• 🧬 Single-nucleotide Resolution: Each nucleotide occupies one token position, preserving high-resolution and position-aligned representations across complete RNA sequences.

• 🧠 Dense Bidirectional Transformer: Every layer uses all-to-all self-attention, allowing each nucleotide to directly access upstream, downstream, and distal sequence context.

• 📏 Native 10K Pretraining: The long-context RiboSpan models are natively pretrained at 10,240 nt, learning transcript-scale context during pretraining rather than relying on inference-time context extension.

• 🧩 Robust High-Masking Representations: Continued pretraining with 40% masking substantially improves reconstruction under heavy corruption while preserving backbone representation quality established during 15% MLM pretraining.

• 🏆 SOTA Frozen Representations: Without task-specific heads or fine-tuning, RiboSpan-10K achieves state-of-the-art frozen RNA representation quality among the evaluated models, with particularly strong performance on long RNAs.

• 🏆 Strongest RNA Encoder Foundation Model: RiboSpan has the strongest RNA understanding capability among RNA encoder foundation models, covering property prediction, functional annotation, and mutation-effect scoring.

🚀 Quick Start

The RIBOSPAN code and inference utilities are maintained in the official GitHub repository, while pretrained model weights are distributed through Hugging Face.

Installation

git clone https://github.com/GAIR-NLP/RIBOSPAN-FM.git
cd RIBOSPAN-FM
pip install -e .

Loading Pretrained Weights

Public RIBOSPAN checkpoints can be loaded directly from Hugging Face:

from pathlib import Path

import torch
import ribospan
from ribospan import RiboSpanForMaskedLM, RiboSpanTokenizer

ckpt = "SII-GAIR-NLP/RIBOSPAN-10K-15"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = RiboSpanTokenizer(str(Path(ribospan.__file__).with_name("vocab.txt")))

model = RiboSpanForMaskedLM.from_pretrained(ckpt).eval().to(device)

from_pretrained() automatically downloads the checkpoint from Hugging Face and caches it locally, so no manual model-weight placement is required. Alternatively, the checkpoint can be downloaded manually to a user-specified local directory and loaded from that path:

ckpt = "path/to/RIBOSPAN-10K-15"

For restricted-access checkpoints, request access through the corresponding Hugging Face model page. Once access is approved, the checkpoint can be loaded using an authenticated Hugging Face account.

For additional inference examples and usage details, see the official GitHub repository.

🏗️ Model and Checkpoints

RiboSpan uses a 32-layer pre-norm bidirectional Transformer encoder with dense multi-head self-attention and SwiGLU feed-forward networks.

The RIBOSPAN model family currently consists of four matched checkpoints:

The 40% checkpoints continue from the corresponding 15% runs, improving reconstruction robustness under heavy corruption while preserving backbone representation quality.

🧬 Pretraining

The RiboSpan pretraining corpus combines diverse RNA sequences from RNAcentral v26.0 with quality-controlled protein-coding transcripts from Ensembl release 115 and Ensembl Genomes release 62. Ensembl-derived transcripts retain complete CDS and UTR annotations, providing full-transcript examples for learning dependencies across coding and untranslated regions.

After source-specific filtering, sequence normalization, and exact deduplication, the final corpus contains 67.6 million RNA sequences comprising 85.7 billion nucleotide tokens.

The long-context models are pretrained with masked language modeling at a native context length of 10,240 nt. Training begins with 15% masking and is subsequently continued under 40% masking. Matched 1K models use the same corpus and backbone architecture with a native context length of 1,024 nt.

📊 Evaluation

We evaluate RiboSpan from four complementary perspectives: nucleotide reconstruction, controlled long-context representation behavior, frozen RNA-type representation quality, and downstream biological prediction.

🧩 Long-Context Reconstruction

Native 10K RiboSpan models maintain strong nucleotide reconstruction at 10,240 nt, whereas short-context models degrade substantially when extrapolated far beyond their pretrained context. Continued pretraining with 40% masking further improves reconstruction under heavy corruption while remaining closely matched to the 15% checkpoint under standard masking.

🔭 Long-Context Representation

We introduce a controlled long-context representation benchmark using complete mRNAs spanning 1,024 to 10,240 nt. A short central interval is rearranged while preserving its nucleotide composition, allowing contextual responsiveness, region-specific differentiation, and the propagation of representation changes to be evaluated jointly.

Native 10K RiboSpan exhibits a distinct long-context representation profile. Compared with short-context dense encoders extrapolated beyond their pretrained range, it preserves substantially stronger contextual organization at transcript-scale lengths. Compared with long-sequence models based on hybrid sequence-mixing architectures, it shows stronger context-dependent differentiation while maintaining highly controlled distal propagation.

Together, these results show that native long-context pretraining enables dense bidirectional attention to combine interaction flexibility with long-range calibration, supporting strong context-dependent reorganization while suppressing broad, non-selective propagation across the transcript.

🗺️ Frozen RNA-Type Representation

Final-layer hidden states are mean-pooled into sequence representations and evaluated directly using leave-one-out cosine 10-NN label recovery and neighborhood purity, with no classifier, projection head, or downstream fine-tuning.

RiboSpan-10K achieves state-of-the-art frozen RNA representation quality among the evaluated foundation models, including the strongest Overall Biotype and Long RNA results. Its advantage is particularly clear for RNAs longer than 1,024 nt, where native long-context modeling preserves substantially more complete sequence information. The 15% and 40% 10K checkpoints remain closely matched, showing that high-masking continued pretraining preserves the learned representation structure.

🧬 Downstream Biological Benchmarks

We evaluate frozen RiboSpan representations on two systematic biological benchmarks of downstream RNA tasks, covering full-transcript property prediction and zero-shot fitness scoring.

Frozen last-layer mean-pooled representations are evaluated with linear probes on mRNABench, covering half-life, mean ribosome load, eCLIP binding, GO, variant effect, and translation efficiency. Separately, the pretrained MLM head is used for zero-shot masked-marginal scoring on RNAGym fitness assays, with no probe and no fine-tuning.

On full-transcript property prediction, RiboSpan-10K-15 is the strongest overall among the evaluated models, and RiboSpan-10K-40 remains closely matched.

On zero-shot fitness scoring, all RiboSpan checkpoints outperform the other evaluated foundation models. The 1K-40 checkpoint is strongest in aggregate, showing that local substitution likelihoods benefit from short-context, high-masking MLM, whereas native 10K pretraining is more decisive for full-transcript prediction.

Each axis is independently scaled to the best of the six models shown; unnormalized scores are reported in the paper.

📄 License

The RIBOSPAN model weights are provided for non-commercial research use under the RIBOSPAN Non-Commercial Model License 1.0.

🙏 Acknowledgements

We thank RNAcentral, Ensembl, and Ensembl Genomes for providing the sequence resources used to construct the RiboSpan pretraining corpus.

We also acknowledge Megatron-LM and Hugging Face Transformers, whose open-source infrastructure supported the development and implementation of RiboSpan.

📖 Citation

If you find RiboSpan useful in your research, please cite:

@misc{wang2026ribospan,
  title         = {{RIBOSPAN}: A Long-Context {RNA} Foundation Model for Versatile {RNA} Modeling},
  author        = {Wang, Ziyuan and Tang, Bohao and Zhang, Fei and Han, Shuo and Liu, Pengfei},
  year          = {2026},
  eprint        = {2608.22849},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

---

RiboSpan — Single-nucleotide Pretraining Across Native-context.