Downloads · 30 days
17
17% of all-time downloads
BERT-updated
Taykhoom/BERT-updated
BERT-updated is a feature extraction model from Taykhoom. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as apache-2.0.
Standard BERT architecture with flashattention2 and sdpa support added.
All-time downloads
100
Public
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.py22.2 KB · 81%
At a glance
- Task
- Feature Extraction
- Library
- transformers
- License
- apache-2.0
- Model type
- bert_updated
- Access
- Public
- Created
- Jun 3, 2026
- Updated
- Aug 26, 2026
- SHA
- c2ee7073
Model card
From the Hugging Face model README
BERT-updated
Standard BERT architecture with flash_attention_2 and sdpa support added.
This is a shared code repository — it contains no pretrained weights. It is used as the code backend for biological sequence models that share the vanilla BERT architecture (post-LN transformer, learned absolute position embeddings) but have model-specific vocabularies and hyperparameters:
- Taykhoom/RNABERT
- Taykhoom/UTRBERT-3mer, 4mer, 5mer, 6mer
- Taykhoom/DNABERT-3mer, 4mer, 5mer, 6mer
- Taykhoom/CodonBERT
- Taykhoom/SpliceBERT-510nt, human-510nt, 1024nt
Each of those repos stores weights, tokenizer, and config; their auto_map in
config.json points here for the modeling code.
What was changed from stock transformers.BertModel
The standard HF BertModel (transformers 4.57.6) supports sdpa but not
flash_attention_2. This repo adds a complete attn_implementation dispatch:
| Backend | Class | Notes |
|---|---|---|
eager | BertSelfAttention | Standard scaled dot-product, identical to original BERT |
sdpa | BertSdpaSelfAttention | F.scaled_dot_product_attention, bool mask -> additive float mask |
flash_attention_2 | BertFlashSelfAttention | flash_attn_varlen_func for padded inputs, flash_attn_func for unpadded |
The rest of the architecture (embeddings, FFN, pooler, weight layout) is unchanged.
Related Models
See the full BERT-updated collection.
| Model | Purpose |
|---|---|
| BERT-updated | Shared BERT runtime |
| MosaicBERT-updated | Shared MosaicBERT runtime |
Usage
Do not load this repo directly. Load one of the model repos listed above:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Taykhoom/RNABERT", trust_remote_code=True)
model = AutoModel.from_pretrained("Taykhoom/RNABERT", trust_remote_code=True)
# Flash Attention 2
model = AutoModel.from_pretrained("Taykhoom/UTRBERT-3mer", trust_remote_code=True,
attn_implementation="flash_attention_2")
Credits
manually by Taykhoom Dalal.
License
Apache 2.0.
- Task
- Feature Extraction
- Library
- transformers
- Type
- bert_updated
- License
- apache-2.0
- Created
- Jun 3, 2026
- Updated
- Aug 26, 2026