Downloads · 30 days
12
20% of all-time downloads
jmvcoelho/Qwen2.5-0.5B-retriever-gpt-dpo-1M
Qwen2.5-0.5B-retriever-gpt-dpo-1M is a feature extraction model from jmvcoelho. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as apache-2.0.
Qwen2.5-0.5B dense retriever (bi-encoder, bidirectional attention, mean pooling), trained on synthetic ClueWeb22 queries produced by a query generator aligned via Direct Preference Optimization (DPO) using a GPT list-…
Downloads · 30 days
12
20% of all-time downloads
All-time downloads
59
Public
Parameters
494M
999 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors988 MB · 98%
From the Hugging Face model README
Qwen2.5-0.5B dense retriever (bi-encoder, bidirectional attention, mean pooling), trained on synthetic ClueWeb22 queries produced by a query generator aligned via Direct Preference Optimization (DPO) using a GPT list-wise reward (GPT-3.5).
Base model: jmvcoelho/Qwen2.5-0.5B-bidirectional-attn-mntp.
| MARCO Doc MRR@100 | MARCO Passage nDCG@10 | TREC-DL19 nDCG@10 | TREC-DL20 nDCG@10 |
|---|---|---|---|
| 0.3795 | 0.3340 | 0.6213 | 0.5496 |
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
repo = "jmvcoelho/Qwen2.5-0.5B-retriever-gpt-dpo-1M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
def encode(texts):
batch = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
out = model(**batch).last_hidden_state
mask = batch["attention_mask"].unsqueeze(-1).float()
emb = (out * mask).sum(1) / mask.sum(1) # mean pooling
return F.normalize(emb, dim=-1) # L2 normalize; score = cosine / dot
q = encode(["what is dense retrieval"])
d = encode(["Dense retrieval uses dual encoders ..."])
print((q @ d.T).item())