ru-promptriever-qwen3-4b

ru-Promptriever-Qwen3-4B

Full Benchmark Comparison

Compare ru-Promptriever with Promptriever, Qwen3-Embedding, multilingual-E5, BGE-M3, BM25, TART-Contriever, INSTRUCTOR-XL, E5-Mistral, and RepLLaMA. The best available score in each metric column is bold. Higher is better.

Each column names its benchmark and metric. p-MRR is shown in percentage points; other metrics use the 0–1 scale. Scroll horizontally to see all columns, or download the table as CSV.

Results combine paper-reported baselines with saved reruns. Qwen3-Embedding-4B uses the values reported in the paper. Empty cells indicate an unreported score or an incompatible evaluation protocol; legacy FollowIR MAP/nDCG values with the older qrels mapping are omitted. Updated mFollowIR-RU and FollowIR reruns use MTEB 2.10.5 and original-instruction judgments. Evaluation records identify each source.

Overview

Standard dense retrieval models score query–passage pairs using a single semantic similarity signal, giving users no control over what "relevant" means beyond keyword choice. Promptriever (Weller et al., 2024) introduced per-instance natural language instructions that dynamically redefine relevance — a capability previously limited to generative LLMs.

ru-Promptriever extends this paradigm to Russian:

• Architecture: Qwen3-based causal LM fine-tuned as a bi-encoder with LoRA + GradCache
• Pooling: last-token (EOS) pooling
• Key training signal: instruction negatives — passages that are topically relevant to the query but violate the instruction constraint

Model Family

---

This Model

This is the final, best-performing model in the ru-Promptriever family. Starting from ru-Promptriever-4B-pretrained, it was further fine-tuned on a balanced multilingual mix of:

• Russian real data — MIRACL + MrTyDi (~11k pairs)
• Russian synthetic data — instruction-augmented and standard pairs (~30k)
• English instruction-following data from the original Promptriever / FollowIR dataset (~24k, ~35% of mix)

Under the current harmonized evaluation protocol, this model obtains 17.28 p-MRR on mFollowIR-RU while matching Promptriever-8B's retrieval quality at half the parameter count.

---

Usage

Basic Retrieval (no instruction)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch.nn.functional as F

model_name = "Vladimirlv/ru-promptriever-qwen3-4b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

def encode(texts: list[str], max_length: int = 512) -> torch.Tensor:
    """Encode texts using last-token (EOS) pooling."""
    inputs = tokenizer(
        texts,
        padding=True,
        truncation=True,
        max_length=max_length,
        return_tensors="pt",
    ).to(model.device)

    with torch.no_grad():
        # Bypass lm_head to get post-norm hidden states
        original_lm_head = model.lm_head
        model.lm_head = torch.nn.Identity()
        outputs = model(**inputs, use_cache=False, return_dict=True)
        model.lm_head = original_lm_head

    # EOS pooling: take embedding at last non-padding token
    seq_len = inputs["attention_mask"].sum(dim=1) - 1
    embeddings = outputs.logits[torch.arange(len(texts)), seq_len]
    return F.normalize(embeddings, p=2, dim=1)


query = "Когда была основана Москва?"
passages = [
    "Москва была основана в 1147 году князем Юрием Долгоруким.",
    "Санкт-Петербург был основан Петром I в 1703 году.",
]

q_emb = encode([query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
print(scores)  # tensor([0.82, 0.61])

Instruction-Following Retrieval

# Append the instruction directly to the query (same format as training)
instruction = "Найди документ, в котором упоминается конкретная дата основания города."
instructed_query = f"{query} {instruction}"

q_emb = encode([instructed_query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
# The model adjusts rankings based on the instruction

Using with sentence-transformers

This model is not compatible with sentence-transformers out of the box due to the custom EOS pooling. Use the snippet above directly with transformers.

---

Model Details

---

Training Data

This model was trained in two stages:

Stage 1 — Pretraining (ru-Promptriever-4B-pretrained): LoRA fine-tuning of Qwen3-4B on ~500k synthetic instruction-augmented Russian retrieval triples from ru-promptriever-dataset, built on top of mMARCO-ru.

Stage 2 — Continued training (this model): further LoRA fine-tuning on a balanced multilingual mix of ~65k rows:

1. Russian real data (~11k) — from MIRACL and MrTyDi retrieval datasets
2. Russian synthetic data (~30k) — instruction-augmented and standard pairs from ru-promptriever-dataset
3. English synthetic data (~24k, ~35% of total) — from the original Promptriever / FollowIR training data

Key properties:

• Instruction negatives: passages that are topically relevant but violate the instruction (3 per instructed query)
• Paired rows: each source query has both a standard row and an instructed row to prevent catastrophic forgetting

---

Intended Use

• Instruction-following dense retrieval in Russian: RAG pipelines, search systems, and scenarios requiring fine-grained query control via natural language
• Research on multilingual instruction-following retrieval and bi-encoder training
• Benchmarking alongside mE5, BGE-M3, and Promptriever-style models

Out-of-Scope

• General-purpose text embedding (use mE5-large or BGE-M3 if no instruction-following is needed)
• Commercial applications (see License below)

---

Limitations

• MS MARCO origin: The training corpus derives from English web passages machine-translated to Russian. A portion of passages retain translation artifacts despite LLM-based rewriting.
• Standard retrieval trade-off: Instruction-following training slightly reduces standard retrieval quality compared to encoder-only models (mE5-large, BGE-M3).
• Synthetic-data quality: Four independent non-author Russian-speaking annotators assessed 64 final instances, with two annotators assigned to the first 32 and two to the remaining 32, giving two independent judgments per instance. Corrected aggregate counts show 95.3% positive-document compliance (122/128 judgments) and 85.2% synthetic-negative rejection (283/332 judgments) under the complete query-instruction condition. This sampled audit characterizes dataset quality; individual synthetic examples may still require filtering for a particular application.

---

License

This model is released under CC BY-NC 4.0 (Creative Commons Attribution–NonCommercial 4.0 International).

The non-commercial restriction is inherited from the upstream MS MARCO license (Microsoft Research License — non-commercial use only), which governs the training corpus.

---

Citation

If you use this model, please cite the original Promptriever paper:

@article{weller2024promptriever,
  title   = {Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
  author  = {Weller, Orion and Van Durme, Benjamin and Lawrie, Dawn and
             Paranjape, Ashwin and Zhang, Yuhao and Hessel, Jack},
  journal = {arXiv preprint arXiv:2409.11136},
  year    = {2024}
}