qwen3-embedding-0.6b-swedish-superlim

qwen3-embedding-0.6b-swedish-superlim

Swedish fine-tune (weight soup) of Qwen/Qwen3-Embedding-0.6B. Beats KBLab/sentence-bert-swedish-cased on both SuperLim-2 tasks that card published: SweParaphrase (STS) and SweFAQ (in-category retrieval).

Related: oxfrug/bge-m3-swedish-superlim — stronger Swedish retrieval lift on the same protocol.

Model details

One file, one forward — not an ensemble. A 768-d cut loses the dual SuperLim win (@768 is 0.821 / 0.578).

Quickstart

pip install -U "sentence-transformers>=2.7" "transformers>=4.51"

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("oxfrug/qwen3-embedding-0.6b-swedish-superlim")
emb = model.encode(["Ett plan lyfter.", "Ett flygplan lyfter."], normalize_embeddings=True)
print(float(emb[0] @ emb[1]))

Retrieval — official Qwen query prompt on the question only:

q = model.encode(["När får jag föräldrapenning?"], prompt_name="query", normalize_embeddings=True)
d = model.encode(["Du kan ansöka på Mina sidor."], normalize_embeddings=True)
print(float(q[0] @ d[0]))

STS / clustering: no prompt. Keep all 1024 dims.

Benchmarks

SuperLim-2 (held-out test)

Spearman of cosine vs human 0–5 on SweParaphrase (n=1378). SweFAQ is in-category accuracy (n=109). Same frozen sbx/superlim-2 splits as the KBLab re-run.

SweParaphrase test was never in train. SweFAQ used official train only — do not call SweFAQ zero-shot. Dev also clears (0.848 / 0.645).

MTEB (Scandinavian, v1)

mteb 2.19.3, max_seq_length=512. 27/28 tasks (DKHateClassification gated, skipped). SweFAQ here is nDCG@10 and is supervised for this soup. SuperLim-targeted checkpoint — it does not beat KBLab on the Swedish-only MTEB mean.

Swedish tasks:

† Supervised for this soup. SwednRetrieval has no train overlap and went the wrong way vs both KBLab and vanilla Qwen.

Nordic category means (27 tasks):

The soup slightly lifts vanilla Qwen on the Nordic mean. It still trails KBLab on the Swedish subset. Use the BGE sister model if you want the broader Swedish retrieval lift.

Raw dumps: superlim/test.json, mteb/MTEB_Scandinavian_v1/. These files document the run; they do not put the model on the official MTEB leaderboard.

How it was made

Weight soup of two Swedish fine-tunes of Qwen/Qwen3-Embedding-0.6B:

0.15 × CoSENT (STS) + 0.85 × CoSENT-then-FAQ

1. CoSENT on all graded SweParaphrase-train pairs (label / 5). SuperLim dev/test blocked. First 0.6B Qwen in this lab over 0.821 Spearman.
2. Light MNRL continue on SweFAQ-train + Swedish healthcare Q–A (Qwen query prefix). FAQ recovered; STS fell to 0.8205.
3. Soup the two state_dicts. Mix weights that clear both bars on dev also clear test.

Do not 4-bit this checkpoint if you want the STS claim. The lead over KBLab is +0.002.

Intended use

Swedish STS, clustering, and retrieval (FAQ / search). Not a general “best Nordic embedder.”

Limits

• Fine-tune + soup, not a new architecture.
• SweFAQ n=109; this is +3 items vs the KBLab re-run.
• Behind the BGE Swedish mix on both SuperLim tasks.
• SuperLim claim is SweParaphrase + SweFAQ only. On MTEB Scandinavian the Swedish-only mean is still under KBLab.

License

Apache-2.0, same as Qwen3-Embedding-0.6B. Fine-tune by oxfrug.

Citation

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}

@inproceedings{berdicevskis-etal-2023-superlim,
  title = {Superlim: A {S}wedish Language Understanding Evaluation Benchmark},
  author = {Berdicevskis, Aleksandrs and Bouma, Gerlof and Kurtz, Robin and Morger, Felix and {\"O}hman, Joey and Adesam, Yvonne and Borin, Lars and Dann{\'e}lls, Dana and Forsberg, Markus and Isbister, Tim and Lindahl, Anna and Malmsten, Martin and Rekathati, Faton and Sahlgren, Magnus and Volodina, Elena and B{\"o}rjeson, Love and Hengchen, Simon and Tahmasebi, Nina},
  booktitle = {Proceedings of EMNLP 2023},
  year = {2023}
}