ImageNet-Small-AttentionOnly

ImageNet Results

In our ImageNet experiment, we aimed to assess the performance of Mice ViTs on a more complex and diverse dataset, ImageNet. We trained mice ViTs on classifying the 1000 ImageNet classes.

Training Details

Similar to the dSprites experiment, for each attention layer setting, we explored two model variants: an attention-only model and a model combining attention with the MLP module. Dropout and layer normalization were not applied for simplicity. The detailed training logs and metrics can be found here.

Table of Results

Below table describe the accuracy [ <Acc> | <Top5 Acc> ] of Mice ViTs with different configurations.