searchlm-nl2bm25-sft

SearchLM NL2BM25 — SFT v1 (Qwen2.5-3B-Instruct)

Part of the SearchLM collection · GitHub

A Qwen2.5-3B-Instruct model fine-tuned via LoRA SFT to convert natural language queries into Tantivy boolean search queries with explicit chain-of-thought reasoning. This is the warm-start checkpoint before GRPO reinforcement learning.

Pipeline position: base → SFT v1 → GRPO v1 (⚠️ reward hacking) → SFT v2 → GRPO v2 ✅

For the best retrieval model, use GRPO v2.

---

What it does

The model outputs a structured two-part response for any natural language information need:

Input:

Do statins cause breast cancer?

Output:

<reasoning>
Key concepts:
1. Statin drugs — synonyms: statin, HMG-CoA reductase inhibitor, simvastatin, atorvastatin,
   lovastatin, pravastatin, rosuvastatin
2. Causal relationship — cause, risk, association, induce, increase risk
3. Breast cancer — "breast cancer", "breast carcinoma", "breast neoplasm", "mammary carcinoma"

Strategy: AND the three concept groups; OR synonyms within each group.
Phrase-quote multi-word terms to prevent term splitting.
</reasoning>
<query>(statin OR "HMG-CoA reductase inhibitor" OR simvastatin OR atorvastatin OR lovastatin)
AND (cause OR risk OR association OR "induce" OR "increase risk")
AND ("breast cancer" OR "breast carcinoma" OR "breast neoplasm")</query>

The <query> block is valid Tantivy boolean syntax ready to pass directly to a search engine.

---

All SearchLM checkpoints

Evaluated on BEIR test splits (NFCorpus: 323 queries, SciFact: 300 queries).

SFT v1 scores slightly below base on NFCorpus and well below on SciFact. The ~36% of training examples with ndcg_at_10 = 0 taught syntactically correct but semantically wrong boolean structure — queries that parsed fine but retrieved nothing. SFT v2 fixes this with a quality filter.

---

Training Details

Training data distribution

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Supreeth/searchlm-nl2bm25-sft",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Supreeth/searchlm-nl2bm25-sft")

SYSTEM_PROMPT = """You are an expert information retrieval specialist. Convert the \
natural language query into a Tantivy boolean search query.

Output format (strictly follow this):
<reasoning>
Step-by-step concept extraction and synonym expansion.
</reasoning>
<query>your boolean query here</query>"""

nl_query = "effects of climate change on coral reef ecosystems"
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Convert to a Tantivy boolean search query:\n\n{nl_query}"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

Tantivy Boolean Syntax

Tantivy is a full-text search engine library. The model targets its query language:

---

Related resources

• Dataset: Supreeth/nl2bm25-sft
• Code: SupreethRao99/searchLM
• Analysis: Reward hacking report
• Collection: SearchLM collection

Citation

@misc{searchlm2026,
  title  = {SearchLM: Training Small Language Models for Boolean Query Generation via RLVR},
  author = {Rao, Supreeth},
  year   = {2026},
  url    = {https://github.com/SupreethRao99/searchLM},
}