retrieval-a025

retrieval-a025 — Coathink writing-recall embedder

A LoRA fine-tune of Qwen/Qwen3-Embedding-0.6B, merged back into the base at α = 0.25 (WiSE-FT style interpolation), for one task: given a sentence a writer is drafting, retrieve the note card (saved highlight) it draws on.

model.safetensors is 1.1 GB, fp16, 1024-dim, Qwen3Model. Standard SentenceTransformers layout — SentenceTransformer("<repo-id>") just works.

⚠️ Read this before choosing it

On our only human-judged benchmark, this model does not beat the frozen off-the-shelf models it was meant to improve on.

n = 34 queries / 859 cards, one neuroscience paper labeled by its own author. Paired MDE ≈ 0.030, so the top four are a statistical tie — but a tie is the honest reading, not a win. A separate agent-labeled ruler scores this model resolvably above base (+0.026), but a cross-ruler agreement test showed that ruler disagrees with the human anchor on exactly this kind of close call, so we do not count it.

Do not use the custom Instruct: ... query prefix. A prompt ablation found it is net-negative for the base model; frozen base with no prompt is the best number in the table above. If you use Octen, keep its native prompt.

Where it does win: mid-sentence queries

Retrieval in the product fires when the writer pauses mid-sentence, not on a finished sentence. Evaluated in that regime (clause truncated to 40/60% with the preceding ~30 words prepended), the ranking flips and this model leads:

This is the only claim we make for this model, and it is a weak one: on a second, cross-domain ruler (a physics/CS paper, 31 queries / 1101 cards) the advantage did not replicate — a025 and base both scored 0.1795 at 40%. Treat the mid-sentence edge as unconfirmed outside the domain it was measured in.

Training

MarginMSE on citation-grounded pairs from unarXive, teacher = Qwen3-Reranker-4B log-odds margins, LoRA r16/α32, then merged at α=0.25. The interpolation is what made it survive out-of-domain; the pure-distill checkpoint (v2a, 0.303) is worse than the base it started from.

Scaling this recipe fails. A 60k field-balanced set of s2orc citation pairs (12 fields, hard negatives, same teacher, same loss) produced 0.2475 from base and 0.2410 continuing from this model — a statistically significant regression (per-query AUC 0.866 vs base 0.934, gap 0.068 > MDE 0.041). The citation-proxy signal is misaligned with human writing-utility judgments; more of it does not help.

Recommendation

For a fresh integration, prefer Octen/Octen-Embedding-0.6B frozen: tied-best or better on both rulers, Apache-2.0, 600 MB, same backbone and MLX path, and no LoRA-merge / prompt-calibration apparatus to maintain. Reach for this model only if you are specifically working the mid-sentence regime and want to reproduce the table above.

No MLX build exists. a025-mlx in the source repo is a broken stub (the safetensors entry is an 84-byte symlink). A Swift/MLX consumer needs a real conversion first; mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ is the only ready-made MLX option today, and 4-bit was measured to cost nothing (device goldset: 4-bit base 0.733 vs bf16 a025 0.730).