roBERTa-nanobeir

TL;DR

This is a roberta-base model I fine-tuned on the NanoBEIR dataset for semantic search and passage retrieval. It's not chasing the top of any leaderboard — it's a compact, fast-loading embedding model that runs happily on modest hardware and still retrieves sensibly.

Why I made this

Most retrieval models on the Hub are enormous. Great results, sure — but they need serious GPUs just to load. I wanted something that:

• fits comfortably on a laptop or a small GPU,
• loads in seconds, not minutes,
• and still generalises across different kinds of retrieval tasks.

NanoBEIR felt like the right training signal for that. It's a compact collection of retrieval tasks that mirrors the diversity of the full BEIR benchmark, so the model gets exposed to a bit of everything — questions, abstracts, web-style queries — without me needing a compute cluster to train it.

How to use it

The easy way, with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("YOUR_USERNAME/roberta-nanobeir")

queries = ["what causes the seasons"]
passages = [
    "The seasons are caused by the tilt of the Earth's rotational axis relative to its orbit around the Sun.",
    "To fix a leaky tap, first shut off the water supply under the sink.",
]

query_emb = model.encode(queries)
passage_emb = model.encode(passages)

scores = model.similarity(query_emb, passage_emb)
print(scores)  # the first passage should win, comfortably

If you'd rather stay in plain transformers, mean pooling + normalisation works fine too:

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/roberta-nanobeir")
model = AutoModel.from_pretrained("YOUR_USERNAME/roberta-nanobeir")

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True,
                       max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs)
    mask = inputs["attention_mask"].unsqueeze(-1)
    emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1)
    return torch.nn.functional.normalize(emb, p=2, dim=1)

Training, in plain words

Nothing exotic here — a pretty standard contrastive setup: | Batch size | 4 | | Learning rate | 2e-5 |

The idea is simple: pull the matching passage close to its query in embedding space, push everything else in the batch away. Repeat a few hundred thousand times and the model slowly learns what "relevant" means across a bunch of different domains.

What it's good at

• First-stage retrieval / candidate search over passages
• Semantic search in apps, docs, RAG pipelines
• Clustering, deduplication and similarity scoring
• Running somewhere bigger models can't (laptops, CPUs, small GPUs)

What it's not good at

• Anything non-English — the training data and base model are English-only
• Very long documents. It inherits RoBERTa's 512-token limit, so if your "passage" is a chapter of a book, the model only ever sees the beginning. Chunk your documents.
• Generative tasks. It's an encoder — it gives you vectors, not text.

Limitations & biases

A few things you should know before using this in anything real:

• The training data is a mix of web-derived corpora, so the embeddings carry the usual baggage of web text. Like any embedding model, it can bake societal biases into its similarity scores. Please test it on your own data before deploying it anywhere that affects people.
• It will occasionally tell you two passages are similar just because they share keywords, not because they mean the same thing. Keyword overlap is a habit every retrieval model has to unlearn, and this one only partly did.
• Small model, small dataset: expect it to struggle on niche domains it never saw during training.

License

The weights are released under the MIT license (inherited from roberta-base). Note that NanoBEIR is assembled from several source corpora with their own licenses — if you redistribute training data or build something commercial on top, it's worth checking the constituent dataset licenses yourself.

Feedback

If this model is useful to you, or if you find a task where it embarrasses itself, open a discussion on the model card — I'd genuinely like to know. Retrieval models improve through people reporting where they fail, not through vibes.