Downloads · 30 days
36
2% of all-time downloads
AntoineBlanot/flan-t5-xxl-classif-3way
flan-t5-xxl-classif-3way is a zero-shot classification model from AntoineBlanot. Use it when you need labels you did not train the model on. It is set up for transformers.
T5ForSequenceClassification adapts the original T5 architecture for sequence classification tasks.
Downloads · 30 days
36
2% of all-time downloads
All-time downloads
2.2K
Public
Repo size
41.2 GB
Likes
4
Public
Click a slice to open those files.
.bin20.6 GB · 100%
From the Hugging Face model README
T5ForSequenceClassification adapts the original T5 architecture for sequence classification tasks.
T5 was originally built for text-to-text tasks and excels in it. It can handle any NLP task if it has been converted to a text-to-text format, including sequence classification task! You can find here how the original T5 is used for sequence classification task.
Our motivations for building T5ForSequenceClassification is that the full original T5 architecture is not needed for most NLU tasks. Indeed, NLU tasks generally do not require to generate text and thus a large decoder is unnecessary. By removing the decoder we can half the original number of parameters (thus half the computation cost) and efficiently optimize the network for the given task.
T5ForSequenceClassification supports the task of zero-shot classification. It can direclty be used for:
Since the T5ForClassification class is currently not supported by the transformers library, you cannot direclty use this model on the Hub. To use T5ForSequenceClassification, you will have to install additional packages and model weights. You can find instructions here.
Models based on the BERT architecture like RoBERTa and DeBERTa have shown very strong performance on sequence classification task and are still widely used today. However, those models only scale up to ~1.5B parameters (DeBERTa xxlarge) resulting in a limited knowledge compare to bigger models. On the other hand, models based on the T5 architecture scale up to ~11B parameters (t5-xxl) and innovations with this architecture are very recent and keeps improving (mT5, Flan-T5, UL2, Flan-UL2, and probably more...)
T5ForClassification Architecture:
Benefits and Drawbacks:
Results on the validation data of training tasks:
| Dataset | Accuracy | F1 |
|---|---|---|
| MNLI (m) | 0.923 | 0.923 |
| MNLI (mm) | 0.922 | 0.922 |
| SNLI | 0.942 | 0.942 |
| SciTail | 0.966 | 0.647 |
Results on validation data of unseen tasks (zero-shot):
| Dataset | Accuracy | F1 |
|---|---|---|
| ? | ? | ? |
Special thanks to philschmid for making a Flan-T5-xxl checkpoint in fp16.