Downloads · 30 days
0
protein-aa-fast-tokenizer
pszmk/protein-aa-fast-tokenizer
protein-aa-fast-tokenizer is a machine learning model from pszmk. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Fast Rust-backed tokenizer for protein sequences.
Access
Public
Updated Jan 1, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json4.8 KB · 62%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Jan 1, 2026
- Updated
- Jan 1, 2026
- SHA
- 8b6cdbfe
Model card
From the Hugging Face model README
Protein Amino-Acid Fast Tokenizer
Fast Rust-backed tokenizer for protein sequences.
Features
- 1 token = 1 amino acid — character-level tokenization
- Fast Rust backend — efficient processing via HuggingFace Tokenizers
- Transformer-ready — compatible with
AutoTokenizer
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("pszmk/protein-aa-fast-tokenizer")
# Single sequence
tokens = tokenizer("MKTLLILAVAVCSAA")
print(tokens)
# {'input_ids': [2, 16, 14, ...], 'attention_mask': [1, 1, ...]}
# Batch with padding
batch = tokenizer(
["MKTLLILAVAVCSAA", "ACDEFGHIK"],
padding=True,
return_tensors="pt",
)
Vocabulary
| ID | Token | Description |
|---|---|---|
| 0 | <PAD> | Padding |
| 1 | <MASK> | Masked token |
| 2 | <CLS> | Classification / Start |
| 3 | <SEP> | Separator |
| 4 | <EOS> | End of sequence |
| 5 | <UNK> | Unknown |
| 6-25 | A-Y | Standard amino acids |
| 26 | X | Any amino acid |
| 27 | B | Asparagine or Aspartic acid |
| 28 | Z | Glutamine or Glutamic acid |
Template Processing
- Single sequence:
<CLS> SEQUENCE <EOS> - Pair sequences:
<CLS> SEQ_A <SEP> SEQ_B <EOS>
Citation
Part of the LAMP (Latent Anti-Microbial Peptides) project.
- Library
- transformers
- License
- mit
- Languages
- en
- Created
- Jan 1, 2026
- Updated
- Jan 1, 2026