dynamic-minilmv2-L6-H384-squad1.1-int8-static

Model Details: QuaLA-MiniLM

This model is the result of a new approach called QuaLA-MiniLM, which combines knowledge distillation, the length-adaptive transformer (LAT) technique, and low-bit quantization. We expand the Dynamic-TinyBERT approach. This approach trains a single model that can adapt to any inference scenario with a given computational budget, achieving a superior accuracy-efficiency trade-off on the SQuAD1.1 dataset. The authors compare their approach to other efficient methods and find that it achieves up to an x8.8 speedup with less than 1% accuracy loss. They also provide their code publicly on GitHub. The article also discusses other related work in the field, including dynamic transformers and other knowledge distillation approaches.

QuaLA-MiniLM training process

<!-- : # Figure showing QuaLA-MiniLM training process.  -->

To run the model with the best accuracy-efficiency tradeoff per a specific computational budget, we set the length configuration to the best setting found by an evolutionary search to match our computational constraint.

How to use

Code examples coming soon!

import ...
 

Metrics (Model Performance):

Inference performance on the SQuAD1.1 evaluation dataset. For all the length-adaptive (LA) models we show the performance both of running the model without token-dropping, and of running the model in a token-dropping configuration according to the optimal length configuration found to meet our accuracy constraint.

Training and Evaluation Data

Ethical Considerations

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model.

BibTeX entry and citation info