GeneLinguaLM-v5

GeneLinguaLM v5

GeneLinguaLM is a multimodal model that generates natural language descriptions of protein functions from amino acid sequences.

Model Description

GeneLinguaLM bridges protein sequences and natural language through cross-modal learning:

• Protein Encoder: ProtBERT (Rostlab/prot_bert_bfd) - encodes amino acid sequences
• Q-Former: Compresses sequence features into 32 query tokens
• LLM: Mistral-7B-Instruct with LoRA fine-tuning
• Projector: Maps Q-Former outputs to LLM embedding space

Performance

GeneLinguaLM outperforms:

• Zero-shot Mistral-7B by 35% on ROUGE-L (demonstrates cross-modal alignment value)
• BioGPT by 43% on ROUGE-L for protein function description

Usage

from genelinguaLM import GeneLinguaLM

# Load model
model = GeneLinguaLM()

# Describe a protein sequence
sequence = "MALWMRLLPLLALLALWGPDPAAAFVNQHLCGSHLVEALYLVCGERGFFYTPKTRREAEDLQVGQVELGGGPGAGSLQPLALEGSLQKRGIVEQCCTSICSLYQLENYCN"
description = model.describe(sequence)
print(description)
# Output: "Hormone that regulates glucose metabolism and blood sugar levels..."

Example Output

Input: Human Insulin sequence

MALWMRLLPLLALLALWGPDPAAAFVNQHLCGSHLVEALYLVCGERGFFYTPKT...

Output:

Hormone that regulates carbohydrate and lipid metabolism.
Plays a key role in the regulation of glucose levels in the blood...

Training Data

• Source: UniProt (Swiss-Prot reviewed entries)
• Organisms: Human (46%), Mouse, Drosophila, Yeast, E.coli, etc. (54%)
• Size: 52,480 protein-description pairs
• Training: 5 epochs, ~16,000 steps

Architecture

Protein Sequence
      ↓
[ProtBERT] → Sequence Embeddings (1024-dim)
      ↓
[Q-Former] → 32 Query Tokens (768-dim)
      ↓
[Projector] → LLM Embeddings (4096-dim)
      ↓
[Mistral-7B + LoRA] → Natural Language Description

Model Files

• checkpoint_step15732.pt: Main checkpoint (Q-Former + Projector weights)
• lora_step15732/: LoRA adapter for Mistral-7B
• qformer_checkpoint.pt: Q-Former pretrained weights

Requirements

torch>=2.0
transformers>=4.35
peft>=0.6

Citation

@misc{genelinguaLM2024,
  title={GeneLinguaLM: Bridging Protein Sequences and Natural Language},
  author={GeneLinguaLM Team},
  year={2024},
  url={https://github.com/powersimmani/geneLLM}
}

Acknowledgments

• UniProt for protein data
• ProtBERT for sequence encoding
• Mistral AI for the base LLM
• BLIP-2 for Q-Former architecture inspiration

License

Apache 2.0