Downloads · 30 days
76
21% of all-time downloads
multilingual-e5-large-pms-16384
alphaedge-ai/multilingual-e5-large-pms-16384
multilingual-e5-large-pms-16384 is a sentence similarity model from alphaedge-ai. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as mit.
This model is a 42.7% smaller version of intfloat/multilingual-e5-large optimized for 16384 language via vocabulary pruning.
All-time downloads
359
Public
Parameters
321M
1.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.3 GB · 100%
At a glance
- Task
- Sentence Similarity
- Library
- sentence-transformers
- License
- mit
- Model type
- xlm-roberta
- Access
- Public
- Created
- Nov 15, 2025
- Updated
- Nov 15, 2025
- SHA
- 9656dbbb
Base models
Datasets
Model card
From the Hugging Face model README
multilingual-e5-large-pms-16384
This model is a 42.7% smaller version of intfloat/multilingual-e5-large optimized for 16384 language via vocabulary pruning.
Total vocabulary size: 16384 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming
Language Distribution
- pms: 16384 tokens
This pruned model should perform similarly to the original model for 16384 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.
Usage
You can use this model with the Transformers library:
from transformers import AutoModel, AutoTokenizer
model_name = "Lumberjackk/multilingual-e5-large-pms-16384"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
Model Statistics
- Original model size: 559.9M parameters
- Pruned model size: 320.7M parameters
- Size reduction: 42.7%
- Vocabulary reduction: 93.4%
- Task
- Sentence Similarity
- Library
- sentence-transformers
- Type
- xlm-roberta
- License
- mit
- Languages
- pms
- Created
- Nov 15, 2025
- Updated
- Nov 15, 2025