Downloads · 30 days
10
7% of all-time downloads
multilingual-e5-base-pms-32768
alphaedge-ai/multilingual-e5-base-pms-32768
multilingual-e5-base-pms-32768 is a sentence similarity model from alphaedge-ai. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as mit.
This model is a 60.0% smaller version of intfloat/multilingual-e5-base optimized for 32768 language via vocabulary pruning.
All-time downloads
147
Public
Parameters
111M
445 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors445 MB · 99%
At a glance
- Task
- Sentence Similarity
- Library
- sentence-transformers
- License
- mit
- Model type
- xlm-roberta
- Access
- Public
- Created
- Oct 23, 2025
- Updated
- Oct 23, 2025
- SHA
- eb9cc06e
Base models
Datasets
Model card
From the Hugging Face model README
multilingual-e5-base-pms-32768
This model is a 60.0% smaller version of intfloat/multilingual-e5-base optimized for 32768 language via vocabulary pruning.
Total vocabulary size: 32768 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming
Language Distribution
- pms: 32768 tokens
This pruned model should perform similarly to the original model for 32768 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.
Usage
You can use this model with the Transformers library:
from transformers import AutoModel, AutoTokenizer
model_name = "Lumberjackk/multilingual-e5-base-pms-32768"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
Model Statistics
- Original model size: 278.0M parameters
- Pruned model size: 111.2M parameters
- Size reduction: 60.0%
- Vocabulary reduction: 86.9%
- Task
- Sentence Similarity
- Library
- sentence-transformers
- Type
- xlm-roberta
- License
- mit
- Languages
- pms
- Created
- Oct 23, 2025
- Updated
- Oct 23, 2025