AutoDisProxyT-STSB

AutoDisProxyT-STSB for Distilling Massive Neural Networks

AutoDisProxyT is a distilled task-agnostic transformer model that leverages task transfer for learning a small universal model that can be applied to arbitrary tasks and languages as outlined in the paper Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models.

This AutoDisProxyT checkpoint with 7 layers, 160 hidden size, 10 attention heads corresponds to 6.88 million parameters and 0.27G FLOPs.

The following table shows the results on GLUE dev set.

Tested with torch 1.6.0

If you use this checkpoint in your work, please cite:

@article{xu2022autodistil,
  title={AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models},
  author={Xu, Dongkuan and Mukherjee, Subhabrata and Liu, Xiaodong and Dey, Debadeepta and Wang, Wenhui and Zhang, Xiang and Awadallah, Ahmed Hassan and Gao, Jianfeng},
  journal={arXiv preprint arXiv:2201.12507},
  year={2022}
}