Downloads · 30 days
8
19% of all-time downloads
flan-t5-large
ctu-aic/flan-t5-large
flan-t5-large is a feature extraction model from ctu-aic. Use it when you need embeddings to search or compare text. It is set up for transformers.
This model's tokenizer is extended with CS, SK and PL accents using the following code:
All-time downloads
43
Public
Repo size
6 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin3 GB · 100%
At a glance
- Task
- Feature Extraction
- Library
- transformers
- Model type
- t5
- Access
- Public
- Created
- Aug 7, 2023
- Updated
- Aug 7, 2023
- SHA
- 6e995ffb
Model card
From the Hugging Face model README
This model's tokenizer is extended with CS, SK and PL accents using the following code:
from transformers import (
AutoModel,
AutoTokenizer,
)
model_id = "google/flan-t5-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)
accents = "áčďéěíňóřšťúůýž" # CS
accents += "ąćęłńóśźż" # PL
accents += "áäčďéíĺľňóôŕšťúýž" # SK
accents += accents.upper()
accents = set(c for c in accents)
new_tokens = accents - set(tokenizer.vocab.keys())
tokenizer.add_tokens(list(new_tokens))
model.resize_token_embeddings(len(tokenizer))
- Task
- Feature Extraction
- Library
- transformers
- Type
- t5
- Created
- Aug 7, 2023
- Updated
- Aug 7, 2023