Downloads · 30 days
65
20% of all-time downloads
multilingual-e5-small-pms-32768
alphaedge-ai/multilingual-e5-small-pms-32768
multilingual-e5-small-pms-32768 is a sentence similarity model from alphaedge-ai. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as mit.
This model is a 70.9% smaller version of intfloat/multilingual-e5-small optimized for 32768 language via vocabulary pruning.
All-time downloads
327
Public
Parameters
34.2M
137 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors137 MB · 98%
At a glance
- Task
- Sentence Similarity
- Library
- sentence-transformers
- License
- mit
- Model type
- bert
- Access
- Public
- Created
- Oct 22, 2025
- Updated
- Oct 22, 2025
- SHA
- c077b779
Base models
Datasets
Model card
From the Hugging Face model README
multilingual-e5-small-pms-32768
This model is a 70.9% smaller version of intfloat/multilingual-e5-small optimized for 32768 language via vocabulary pruning.
Total vocabulary size: 32768 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming
Language Distribution
- pms: 32768 tokens
This pruned model should perform similarly to the original model for 32768 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.
Usage
You can use this model with the Transformers library:
from transformers import AutoModel, AutoTokenizer
model_name = "Lumberjackk/multilingual-e5-small-pms-32768"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
Model Statistics
- Original model size: 117.7M parameters
- Pruned model size: 34.2M parameters
- Size reduction: 70.9%
- Vocabulary reduction: 86.9%
- Task
- Sentence Similarity
- Library
- sentence-transformers
- Type
- bert
- License
- mit
- Languages
- pms
- Created
- Oct 22, 2025
- Updated
- Oct 22, 2025