Downloads · 30 days
0
Vladimirlv/ru-promptriever-qwen3-4b-ru-only
ru-promptriever-qwen3-4b-ru-only is a feature extraction model from Vladimirlv. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
[](https://arxiv.org/abs/2409.11136) [](https://huggingface.co/datasets/Vladimirlv/ru-promptriever-dataset) [](https://github.com/Vdmrl/ru-promptriever)
Downloads · 30 days
0
Access
Public
Updated Jul 15, 2026
Repo size
9.1 GB
Likes
0
Public
Click a slice to open those files.
.pt265 MB · 48%
From the Hugging Face model README
Standard dense retrieval models score query–passage pairs using a single semantic similarity signal, giving users no control over what "relevant" means beyond keyword choice. Promptriever (Weller et al., 2024) introduced per-instance natural language instructions that dynamically redefine relevance — a capability previously limited to generative LLMs.
ru-Promptriever extends this paradigm to Russian:
| Model | Parameters | Description | Link |
|---|---|---|---|
| ru-Promptriever-4B | 4B | Final model — best results | link |
| ru-Promptriever-4B-pretrained | 4B | Base pretrained on synthetic data only | link |
| ru-Promptriever-4B-ru-only | 4B | Continued training on Russian-only data | this model |
| ru-Promptriever-1.7B | 1.7B | Scaling experiment | link |
| ru-Promptriever-0.6B | 0.6B | Scaling experiment | link |
This is the Russian-only continued training variant. Starting from ru-Promptriever-4B-pretrained (trained on synthetic data only), it was further fine-tuned on a mix of:
Adding real retrieval data (not just synthetic) improved both nDCG and p-MRR compared to the pretrained model. However, the final model with additional English instruction-following data achieves even higher p-MRR.
Note: For best performance, use the final model instead.
Russian split of mFollowIR — multilingual instruction-following retrieval using TREC NeuCLIR narratives as instructions.
p-MRR (Pairwise Mean Reciprocal Rank, ×100) is the primary instruction-following metric — higher means the model correctly adjusts rankings when instructions change. nDCG@20 measures standard retrieval quality.
| Model | nDCG@20 | p-MRR |
|---|---|---|
| BM25 | 0.452 | +0.67 |
| mE5-large | 0.428 | −2.03 |
| BGE-M3 | 0.479 | −4.15 |
| Promptriever-8B | 0.532 | +12.21 |
| Qwen3-Embedding-4B | 0.549 | +8.10 |
| ru-Promptriever-0.6B | 0.231 | −4.35 |
| ru-Promptriever-1.7B | 0.444 | +14.65 |
| ru-Promptriever-4B-pretrained | 0.461 | +15.26 |
| ru-Promptriever-4B-ru-only (this model) | 0.512 | +16.80 |
| ru-Promptriever-4B | 0.512 | +18.57 |
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch.nn.functional as F
model_name = "Vladimirlv/ru-promptriever-qwen3-4b-ru-only"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
def encode(texts: list[str], max_length: int = 512) -> torch.Tensor:
"""Encode texts using last-token (EOS) pooling."""
inputs = tokenizer(
texts,
padding=True,
truncation=True,
max_length=max_length,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
# Bypass lm_head to get post-norm hidden states
original_lm_head = model.lm_head
model.lm_head = torch.nn.Identity()
outputs = model(**inputs, use_cache=False, return_dict=True)
model.lm_head = original_lm_head
# EOS pooling: take embedding at last non-padding token
seq_len = inputs["attention_mask"].sum(dim=1) - 1
embeddings = outputs.logits[torch.arange(len(texts)), seq_len]
return F.normalize(embeddings, p=2, dim=1)
query = "Когда была основана Москва?"
passages = [
"Москва была основана в 1147 году князем Юрием Долгоруким.",
"Санкт-Петербург был основан Петром I в 1703 году.",
]
q_emb = encode([query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
print(scores) # tensor([0.82, 0.61])
# Append the instruction directly to the query (same format as training)
instruction = "Найди документ, в котором упоминается конкретная дата основания города."
instructed_query = f"{query} {instruction}"
q_emb = encode([instructed_query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
# The model adjusts rankings based on the instruction
This model is not compatible with sentence-transformers out of the box due to the custom EOS pooling. Use the snippet above directly with transformers.
| Property | Value |
|---|---|
| Base model | ru-Promptriever-4B-pretrained (continued training) |
| Architecture | CausalLM bi-encoder (EOS pooling) |
| Fine-tuning method | LoRA (rank-32, α=64, all linear layers) |
| Training data | ~28k rows (11k Russian real + 11k synthetic instructed + 6k synthetic standard) |
| Effective batch size | 128 (8 per device × 4 accum × 4 GPUs) |
| Loss | InfoNCE contrastive (temperature=0.01) |
| Learning rate | 5e-5 |
| Epochs | 2 |
| Max query length | 512 tokens |
| Max passage length | 256 tokens |
This model was trained on a mix of:
Key properties:
This model is released under CC BY-NC 4.0 (Creative Commons Attribution–NonCommercial 4.0 International).
The non-commercial restriction is inherited from the upstream MS MARCO license (Microsoft Research License — non-commercial use only), which governs the training corpus.
If you use this model, please cite the original Promptriever paper:
@article{weller2024promptriever,
title = {Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
author = {Weller, Orion and Van Durme, Benjamin and Lawrie, Dawn and
Paranjape, Ashwin and Zhang, Yuhao and Hessel, Jack},
journal = {arXiv preprint arXiv:2409.11136},
year = {2024}
}