Downloads · 30 days
0
perovskite-tokenizer
GoshKolotyan/perovskite-tokenizer
perovskite-tokenizer is a text generation model from GoshKolotyan. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A specialized tokenizer for perovskite chemical formulas trained on 44 chemical elements.
Access
Public
Updated Sep 5, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json10.5 KB · 81%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Access
- Public
- Created
- Sep 5, 2025
- Updated
- Sep 5, 2025
- SHA
- 876d62fb
Try a prompt
Model card
From the Hugging Face model README
Perovskite Chemical Tokenizer
A specialized tokenizer for perovskite chemical formulas trained on 44 chemical elements.
Features
- Chemical-aware tokenization preserving element boundaries
- Trained on large-scale perovskite formula dataset
- Vocabulary size: 169
- Supports 44 chemical elements including organic cations (MA, FA, etc.)
Usage
from tokenizers import Tokenizer
from perovskite_tokenizer import PerovskiteTokenizer
# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
# Tokenize a formula
result = tokenizer.encode("CsPbI3")
print(result.tokens) # ['[CLS]', 'Cs', 'Pb', 'I', '3', '[SEP]']
Elements Supported
DMA, Na, Te, Bi, Sb, Ti, Tb, Mg, Li, Cd, Yb, Ni, Nb, Mn, Cu, Tl, Zn, Br, Sr, Hg...
- Task
- Text Generation
- License
- apache-2.0
- Languages
- en
- Created
- Sep 5, 2025
- Updated
- Sep 5, 2025