Downloads · 30 days
15
14% of all-time downloads
andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir-langmoe
apertus-v1.5-swiss-embed-4.9b-bidir-langmoe is a sentence similarity model from andreasmartin. Use it when you need a score for how close two texts are. It is set up for sentence-transformers. The card lists the license as apache-2.0.
A Sentence Transformers retrieval model adapted from andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir by replacing the parent's final dense embedding projection with a learned sentence-level SparseMixtureOfExperts pr…
Downloads · 30 days
15
14% of all-time downloads
All-time downloads
104
Public
Parameters
4.9B
9.9 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors9.9 GB · 100%
From the Hugging Face model README
A Sentence Transformers retrieval model adapted from
andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir
by replacing the parent's final dense embedding projection with a learned sentence-level
SparseMixtureOfExperts projection.
existing Sentence Transformers embedding parent
↓
all modules before the parent's final Dense
↓
SparseMixtureOfExperts
├── learned linear router
├── softmax
├── Top-2
└── weighted selected expert projections
↓
parent Normalize
This Swiss experiment uses 7 experts labelled
['de', 'en', 'fr', 'it', 'rm', 'gsw', 'shared']. The labels are experimental metadata; the MoE module itself
contains no language-specific logic.
No router temperature is used.
| Property | Value |
|---|---|
| Total parameters | 4,929,328,175 (4.929B) |
| Active parameters / sentence | 4,908,356,655 |
| Sparse MoE head parameters | 29,388,807 |
| Active head parameters / sentence | 8,417,287 |
| Router parameters | 28,679 |
| Parameters / expert | 4,194,304 |
| Experts | 7 |
| Active experts / sentence | 2 |
| Native embedding dimension | 1024 |
| Matryoshka dimensions | [1024, 768, 512, 256] |
| MoE training max length | 512 tokens |
| Inference max length | 1024 tokens |
| Sparse/reference max abs diff | 1.907e-06 |
SparseMixtureOfExperts is currently exported as a repository-local candidate module. Until an
equivalent module is registered upstream in Sentence Transformers, loading this experimental
artifact requires trust_remote_code=True:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir-langmoe",
trust_remote_code=True,
)
queries = model.encode_query([
"Welche Aufgaben hat der Bundesrat?"
])
documents = model.encode_document([
"Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
"Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
])
scores = model.similarity(queries, documents)
print(scores)
If SparseMixtureOfExperts becomes a registered Sentence Transformers module, the same conceptual
architecture can be serialized without repository-local Python code and loaded without
trust_remote_code=True.
| Property | Value |
|---|---|
| Existing embedding parent | andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir |
| Parent revision | 38e8ef280c560d9c871d20e38ebd9e353e2500f0 |
| Parent modules | ['Transformer', 'Pooling', 'Dense', 'Normalize'] |
| Replaced module | final Dense projection |
| Preserved output normalization | yes |
The parent was already an embedding model. This notebook does not create or modify a native backbone-level MoE LLM.
The parent modules before the final Dense projection were frozen and their sentence representations cached. The original Dense projection initialized every expert, so the model starts from the parent's embedding transformation before experts specialize.
The retrieval objective is combined with:
The declared language is not required at inference. Routing is computed from the sentence representation itself.
Training data:
eljuanina/VotingBooklets-Diamond-v1 is held-out evaluation-only.
Actual training set: 31,483 triplets.
See training_metadata.json for the exact training configuration,
router_diagnostics.json for held-out routing counts, and data_manifest.json for immutable
dataset revisions and processed-file hashes.
These metrics are internal development diagnostics, not MTEB/MMTEB benchmark claims.
| Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
|---|---|---|---|---|---|
| Swiss monolingual | 1024 | 87.29% | 95.00% | 0.9109 | 0.8984 |
| Swiss monolingual | 512 | 87.29% | 95.00% | 0.9109 | 0.8985 |
| Swiss cross-lingual | 1024 | 33.75% | 99.38% | 0.7140 | 0.6183 |
| Swiss cross-lingual | 512 | 33.33% | 99.58% | 0.7121 | 0.6152 |
Monolingual evaluation queries: 480
Cross-lingual evaluation queries: 480
Recorded shared-expert Top-k selections in the held-out diagnostic: 15.
Dense -> Normalize.