ProCALM

ProCALM

ProCALM (Protein Conditionally Adapted Language Model) is a suite of models where ProGen2-base is finetuned with conditional adapters for conditional generation of functional enzymes, based on EC number, taxonomy, or both.

ProCALM models share tokenizer.json, and individual models are organized into subfolders. We have uploaded the most relevant models here, but please reach out if you would like to use other models from our paper. 1.5B and 9B refer to checkpoints trained to 1.5 and 9 billion tokens, respectively.

General Usage

Usage details with examples can be found in github under "Generation" and in our paper. Example framework for generation from pretrained models:

from tokenizers import Tokenizer
from model import ProgenConditional

model = ProgenConditional.from_pretrained("jsunn-y/ProCALM", subfolder="ec-onehot-swissprot/1.5B")
tokenizer = Tokenizer.from_pretrained("jsunn-y/ProCALM")

with torch.no_grad():
  input_ids = torch.tensor(self.tokenizer.encode(context).ids).view([1, -1]).to(self.device)
  tokens_batch = model.generate(input_ids=input_ids, condition_encodings=condition_encodings, do_sample=True, temperature=temperature, max_length=max_length, top_p=top_p, num_return_sequences=num_return_sequences, pad_token_id=self.pad_token_id, eos_token_id=4)

as_lists = lambda batch: [batch[i, ...].detach().cpu().numpy().tolist() for i in range(batch.shape[0])]
sequences = tokenizer.decode_batch(as_lists(tokens_batch))

Note that condition_encodings is a representation of the conditioning, which can be calculated using the dictionaries .pt provided in our github under data.

Summary of Available Models