m2v-e5-large-edu

m2v-e5-large-edu — static embeddings for educational content

A fast, torch-free static embedding model tuned for educational metadata. It is a Model2Vec distillation of intfloat/multilingual-e5-large enriched with a domain vocabulary of curated education-domain keyword tags, so that subject terms and multi-word phrases (e.g. "Quadratische Gleichung", "Erneuerbare Energien") are represented as atomic units.

The base model is a multilingual XLM-RoBERTa-large, so the model still embeds many languages; the added domain vocabulary is primarily German, so the strongest gains are on German educational text. Inference is numpy-only — no PyTorch — and fast: in our CPU benchmark it encoded ~7000 texts/s in a single process.

How it was built

• Base model: intfloat/multilingual-e5-large (a strong multilingual retrieval encoder, XLM-RoBERTa-large).
• Distillation: model2vec.distill with mean pooling (e5's native pooling — this choice has a large effect on quality), pca_dims=512, float32, and SIF weighting (sif_coefficient=1e-4, Zipf-based frequency down-weighting of common tokens).
• Domain vocabulary: ~70,000 cleaned keyword tags (document frequency ≥ 3) extracted from the keyword field of ~340k educational resources — mostly German subject terms — added on top of the base subword vocabulary. Subword fallback is kept for out-of-vocabulary text.

Evaluation

Evaluated on a German educational subject-classification task and compared against other embedding models.

Methodology (identical across models — only the embedding model differs): 25,068 educational items with 47 subject labels and a fixed train / validation / held-out test split. Each embedding model is frozen; a small MLP classification head is trained on top (Optuna hyperparameter search on validation), per-label decision thresholds are tuned on validation, and F1 is reported on the held-out test split that is never used for training or tuning (honest, non-optimistic evaluation).

What this shows: among static distillations of strong multilingual encoders, this model lands in the same band (~0.56–0.60 macro) — the practical ceiling for frozen static embeddings on this short, keyword-heavy metadata. A transformer (e5-small) stays slightly ahead, as expected for static embeddings; the trade-off is that this model needs no PyTorch and is far lighter and faster at inference. For this specific classification task a plain TF-IDF + linear classifier scores higher still (macro ≈ 0.62), so the intended value of this model is fast, torch-free semantic embeddings for the education domain, not being the single best classifier for one dataset. These are domain-specific benchmark numbers, not a general-purpose (e.g. MTEB) score.

Intended use

Fast embedding of (mostly German) educational metadata — titles, descriptions, keywords — for classification, clustering, semantic search and retrieval, especially in CPU-only or low-resource deployments where a transformer is too heavy.

Usage

from model2vec import StaticModel

model = StaticModel.from_pretrained("JanSchachtschabel/m2v-e5-large-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])

Or via Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("JanSchachtschabel/m2v-e5-large-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])

How Model2Vec works

Model2Vec passes a vocabulary through a sentence-transformer, reduces dimensionality with PCA, and weights the token embeddings with SIF weighting (frequent tokens down-weighted). At inference it takes the (weighted) mean of the static token vectors of a text — no transformer forward pass.

License

Inherits the base model's license (multilingual-e5-large: MIT).

Citation

@article{minishlab2024model2vec,
  author = {Tulkens, Stephan and {van Dongen}, Thomas},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}