Downloads · 30 days
920
43% of all-time downloads
NorseDrunkenSailor/Qwen_smol_GH114
Qwen_smol_GH114 is a text generation model from NorseDrunkenSailor. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is a Qwen2-style protein language model trained on SaPROT 3Di-aware protein sequences. Unlike SaPROT it is a CLM rather than a MLM, so it's generative (This becomes useful for DPO and the TRL-trainer from HF).
Downloads · 30 days
920
43% of all-time downloads
All-time downloads
2.1K
Public
Parameters
30.7M
61.4 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors61.4 MB · 100%
From the Hugging Face model README
This is a Qwen2-style protein language model trained on SaPROT 3Di-aware protein sequences. Unlike SaPROT it is a CLM rather than a MLM, so it's generative (This becomes useful for DPO and the TRL-trainer from HF).
This model serves as a specialized base model designed for GH114 reinforcement alignment. It captures the structural and sequence properties of glycoside hydrolase family 114 (GH114) enzymes and their structural neighbors.
This model was specifically developed for the AMLD Intelligence Summit 2026 EnzymeML workshop.
The model was pre-trained on the westlake-repl/AF2_UniRef50 dataset. This provides a robust foundation of protein structure-sequence understanding using the SaPROT 3Di alphabet. Batch Size 896 with 512 sequence length @ 10k Steps (smol training run). 4.58 billion-tokens. Final Train Loss 3.3809 Validation Loss 3.4621.
Following pre-training, the model was fine-tuned on a curated dataset of ≈700,000 structural homologs. These homologs were selected based on shared InterPro domains with the GH114 dataset (IPR004352, IPR017853, IPR013785, IPR000254), ensuring the model is highly sensitive to the structural motifs relevant to this specific enzyme family. Anything within 90% sequence identity from the 55 GH114 sequences was removed from the training set. Two validation sets were used concurrently to monitor distribution overfitting (i.i.d) and the out-of-distribution generalization on the homologs of interest. 4k Steps. 896 batch size, 512 max len. Train Loss 1.7648 Validation Loss 1.8568.
You can load this model using the Hugging Face transformers library.
Note: Ensure your input sequences are converted to the 3Di format (Foldseek alphabet) before passing them to the model.
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_name = "NorseDrunkenSailor/Qwen_smol_GH114"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
# Example input (3Di sequence)
sequence = "M#L#HdSdLdLdAdAdSdFdAd"
inputs = tokenizer(sequence, return_tensors="pt")
# Generate continuation or embeddings
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
This model relies on the 3Di alphabet from Foldeek and the SaProt idea of using these concatenated 3Di-sequence tokens in a PLM.
'''bibtex @article{su2023saprot, title={SaProt: Protein Language Modeling with Structure-aware Vocabulary}, author={Su, Jin and Han, Chenchen and Zhou, Yuyang and Shan, Junjie and Zhou, Xibin and Yuan, Fajie}, journal={bioRxiv}, year={2023}, publisher={Cold Spring Harbor Laboratory} }
@article{van2023foldseek, title={Foldseek: fast and accurate protein structure search}, author={van Kempen, Michel and et al.}, journal={Nature Biotechnology}, year={2024} } '''