multilingual-e5-small-ne-pruned

multilingual-e5-small-ne-pruned

This model is a token-embedding pruned version of intfloat/multilingual-e5-small.

Token-embedding pruning clusters semantically similar tokens in the embedding space (using DBSCAN) and merges each cluster into a single shared embedding, shrinking the vocabulary and reducing memory without retraining the transformer layers.

How to use

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jangedoo/multilingual-e5-small-ne-pruned", 
                            trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])

Note: trust_remote_code=True is required because the model ships a small custom tokenizer class (pruned_tokenizer.py) that applies the id remapping after tokenization. No additional package installation is needed.

Pruning statistics

Evaluation

Citation

If you use this model or the pruning approach, please cite:

@misc{subedi2025tokenpruning,
  author = {Sanjaya Subedi},
  title  = {Token Embedding Pruning for Sentence Transformers},
  year   = {2026},
  note   = {Available at: https://sanjayasubedi.com.np/deeplearning/shrinking-embedding-models-by-pruning-vocabulary/}
}