multilingual-e5-base-pms-16384

multilingual-e5-base-pms-16384

This model is a 64.5% smaller version of intfloat/multilingual-e5-base optimized for 16384 language via vocabulary pruning.

Total vocabulary size: 16384 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming

Language Distribution

• pms: 16384 tokens

This pruned model should perform similarly to the original model for 16384 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.

Usage

You can use this model with the Transformers library:

from transformers import AutoModel, AutoTokenizer

model_name = "Lumberjackk/multilingual-e5-base-pms-16384"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Model Statistics

• Original model size: 278.0M parameters
• Pruned model size: 98.6M parameters
• Size reduction: 64.5%
• Vocabulary reduction: 93.4%