m2v-gte-256-edu

m2v-gte-256-edu — compact static embeddings for educational content

A small, fast, torch-free static embedding model tuned for educational metadata — the compact 256-dimensional member of the family, built for small / edge / low-resource deployments. It is a Model2Vec distillation of Alibaba-NLP/gte-multilingual-base enriched with a domain vocabulary of education-domain keyword tags (multi-word subject terms become atomic units).

The base model is multilingual; the added domain vocabulary is primarily German, so the strongest gains are on German educational text. Inference is numpy-only — no PyTorch.

Why this one: at 256 dimensions it is roughly half the size of the 512-d siblings (~321 MB float32, or ~80 MB loaded as int8) while staying within ~0.01 macro-F1 of them on our benchmark — a strong size/quality trade-off for small applications.

How it was built

• Base model: Alibaba-NLP/gte-multilingual-base (a strong, efficient multilingual encoder).
• Distillation: model2vec.distill with mean pooling (empirically required for this base — CLS pooling collapses it), pca_dims=256, float32, SIF weighting (sif_coefficient=1e-4).
• Domain vocabulary: ~70,000 cleaned keyword tags (document frequency ≥ 3) from the keyword field of ~340k educational resources — mostly German subject terms — added on top of the base subword vocabulary; subword fallback kept for out-of-vocabulary text.

Evaluation

German educational subject-classification, honest held-out test (25,068 items, 47 labels; each frozen embedding model + a small Optuna-tuned MLP head + per-label thresholds tuned on validation; F1 on a test split never used for training/tuning).

What this shows: this 256-d model matches the 512-d e5-large sibling on macro-F1 at half the dimensionality, and lands only ~0.01–0.02 below the best static models — the practical ceiling for frozen static embeddings on this short, keyword-heavy metadata is ~0.56–0.60 macro regardless of base/size. A transformer (e5-small) stays slightly ahead. The value here is a very small, fast, torch-free multilingual education embedder. These are domain-specific benchmark numbers, not a general-purpose (e.g. MTEB) score.

Intended use

Fast embedding of (mostly German) educational metadata — titles, descriptions, keywords — for classification, clustering, semantic search and retrieval, especially in CPU-only, edge, or memory-constrained deployments where the 512-d models or a transformer are too heavy.

Usage

from model2vec import StaticModel

model = StaticModel.from_pretrained("JanSchachtschabel/m2v-gte-256-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])

# Even smaller: load quantized to int8 (~80 MB, near-identical quality)
model_int8 = StaticModel.from_pretrained("JanSchachtschabel/m2v-gte-256-edu", quantize_to="int8")

Or via Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("JanSchachtschabel/m2v-gte-256-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])

How Model2Vec works

Model2Vec passes a vocabulary through a sentence-transformer, reduces dimensionality with PCA, and weights the token embeddings with SIF weighting (frequent tokens down-weighted). At inference it takes the (weighted) mean of the static token vectors of a text — no transformer forward pass.

License

Inherits the base model's license (gte-multilingual-base: Apache-2.0).

Citation

@article{minishlab2024model2vec,
  author = {Tulkens, Stephan and {van Dongen}, Thomas},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}