Gecko-110m-en

litert-community/Gecko-110m-en

This model provides a few variants of the embedding model published in the Gecko paper that are ready for deployment on Android or iOS using LiteRT stack or google ai edge RAG SDK.

Use the models

Android

• Try out the gecko embedding model in the google ai edge RAG SDK. You can find the SDK on GitHub or follow our android guide to install directly from Maven. We have also published a sample app.
• Use the sentencepiece model as the tokenizer for the Gecko embedding model.

Performance

Android

Note that all benchmark stats are from a Samsung S23 Ultra.

<table border="1"> <tr> <th></th> <th>Backend</th> <th>Max sequence length</th> <th>Init time (ms)</th> <th>Inference time (ms)</th> <th>Memory (RSS in MB)</th> <th>Model size (MB)</th> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">GPU</p></td> <td><p style="text-align: right">256</p></td> <td><p style="text-align: right">1306.06</p></td> <td><p style="text-align: right">76.2</p></td> <td><p style="text-align: right">604.5</p></td> <td><p style="text-align: right">114</p></td> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">GPU</p></td> <td><p style="text-align: right">512</p></td> <td><p style="text-align: right">1363.38</p></td> <td><p style="text-align: right">173.2</p></td> <td><p style="text-align: right">604.6</p></td> <td><p style="text-align: right">120</p></td> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">GPU</p></td> <td><p style="text-align: right">1024</p></td> <td><p style="text-align: right">1419.87</p></td> <td><p style="text-align: right">397</p></td> <td><p style="text-align: right">871.1</p></td> <td><p style="text-align: right">145</p></td> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">CPU</p></td> <td><p style="text-align: right">256</p></td> <td><p style="text-align: right">11.03</p></td> <td><p style="text-align: right">147.6</p></td> <td><p style="text-align: right">126.3</p></td> <td><p style="text-align: right">114</p></td> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">CPU</p></td> <td><p style="text-align: right">512</p></td> <td><p style="text-align: right">30.04</p></td> <td><p style="text-align: right">353.1</p></td> <td><p style="text-align: right">225.6</p></td> <td><p style="text-align: right">120</p></td> </tr> <tr> <td><p style="text-align: right">dynamic_int8</p></td> <td><p style="text-align: right">CPU</p></td> <td><p style="text-align: right">1024</p></td> <td><p style="text-align: right">79.17</p></td> <td><p style="text-align: right">954</p></td> <td><p style="text-align: right">619.5</p></td> <td><p style="text-align: right">145</p></td> </tr> </table>

• Model Size: measured by the size of the .tflite flatbuffer (serialization format for LiteRT models)
• Memory: indicator of peak RAM usage
• The inference is run on CPU is accelerated via the LiteRT XNNPACK delegate with 4 threads
• The inference on GPU is accelerated via LiteRT GPU delegate.
• Benchmark is done assuming XNNPACK cache is enabled
• dynamic_int8: quantized model with int8 weights and float activations.