potion-multilingual-128M-i8-tokenade

potion-multilingual-128M-i8-tokenade

An int8 quantisation of minishlab/potion-multilingual-128M, plus precomputed tokenizer artifacts for low-memory loading.

All credit for the model itself goes to the Minish Lab — this repo adds no training, no distillation and no new data. It only changes how the same weights are stored and loaded.

Why

model2vec inference is tokenize → look up rows → mean-pool: no context, no forward pass. Loading it should not cost like a transformer, but in practice a stock load decodes the whole weight matrix to f32 and inflates tokenizer.json into an in-memory trie — together about 1 GB of RSS and ~930 ms before the first vector. That is fine for a long-lived server and prohibitive for a short-lived CLI process.

These artifacts are the same model arranged so it can be memory-mapped instead:

The per-embedding cost regresses (no pre-decoded matrix, no cache); a process needs roughly 12 700 embeddings in one run before that outweighs the load-time saving.

Files

Quantisation

Symmetric, one global scale (max|W| / 127). The scale is deliberately not stored: mean-pooling is linear and model2vec L2-normalises its output, so a single global factor cancels exactly. Rows can therefore be read straight from disk as raw i8.

This is measured, not assumed. Against model2vec-rs reading the f32 original, over 100 133 text segments extracted from real PDF / DOCX / XLSX / EPUB / ODT / ODS / ODP / PPTX files:

• token ids: 0 divergences
• embeddings: worst cosine 0.999994

Retrieval quality on our internal evaluation is unchanged within noise (0.570/0.413 vs 0.571/0.414 for f32).

Caveats

• [UNK] handling follows model2vec-rs exactly, including that it does not drop [UNK] rows for a Unigram tokenizer (it looks for a string unk_token field, which Unigram does not emit). If you consume these artifacts with your own loader and drop [UNK] instead, you will get different vectors on unk-heavy text.
• vocab.fst maps duplicate tokens to their last id, matching how tokenizers builds its token_to_ids map.
• The FST/score/norm layout is ours, not a model2vec standard. Anything reading only model.safetensors + tokenizer.json (including model2vec itself) works with this repo unmodified — the i8 tensor is a standard safetensors dtype.

License

MIT, inherited from the upstream model.