SKILLRET-Reranker-0.6B

SkillRet-Reranker-0.6B

This is a reranker fine-tuned for AI agent skill retrieval. Given a natural-language user request and a candidate agent skill, it scores how relevant and useful the skill is for the request. It is designed as the second stage after a first-stage retriever such as SkillRet-Embedding-0.6B or SkillRet-Embedding-8B.

The model is fine-tuned from Qwen/Qwen3-Reranker-0.6B on the SkillRet benchmark training split with binary cross-entropy on the yes/no token probability. It keeps the scoring interface of Qwen3-Reranker.

Usage

Transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "anonymous-ed-benchmark/SKILLRET-Reranker-0.6B"
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).eval()

token_yes = tokenizer.convert_tokens_to_ids("yes")
token_no = tokenizer.convert_tokens_to_ids("no")

instruction = (
    "Given a skill search query, judge whether the skill document "
    "is relevant and useful for the query"
)
prefix = (
    "<|im_start|>system\nJudge whether the Document meets the requirements based on the "
    'Query and the Instruct provided. Note that the answer can only be "yes" or "no".'
    "<|im_end|>\n<|im_start|>user\n"
)
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
prefix_ids = tokenizer.encode(prefix, add_special_tokens=False)
suffix_ids = tokenizer.encode(suffix, add_special_tokens=False)
max_length = 8192


def format_pair(query: str, doc: str) -> str:
    return f"<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}"


@torch.no_grad()
def score(query: str, docs: list[str]) -> list[float]:
    pairs = [format_pair(query, d) for d in docs]
    enc = tokenizer(
        pairs,
        padding=False,
        truncation="longest_first",
        return_attention_mask=False,
        max_length=max_length - len(prefix_ids) - len(suffix_ids),
    )
    enc["input_ids"] = [prefix_ids + ids + suffix_ids for ids in enc["input_ids"]]
    enc = tokenizer.pad(enc, padding=True, return_tensors="pt").to(model.device)
    logits = model(**enc).logits[:, -1, :]
    stacked = torch.stack([logits[:, token_no], logits[:, token_yes]], dim=1)
    return torch.nn.functional.log_softmax(stacked, dim=1)[:, 1].exp().tolist()


query = "Help me set up a CI/CD pipeline for my Python project"
skills = [
    "ci-cd-setup | Configure continuous integration and deployment pipelines ...",
    "python-debugging | Debug Python applications using pdb and logging ...",
]
print(score(query, skills))  # higher = more relevant

Each skill document is formatted as name | description | SKILL.md body, the same representation used by the SkillRet embedding models.

Training Details

• Base model: Qwen3-Reranker-0.6B (0.6B parameters)
• Training data: SkillRet benchmark training split (63,259 queries and 10,123 skills)
• Objective: binary cross-entropy on P(yes) for each query–skill pair
• Hard negatives: mined from four retrievers (SkillRet-Embedding-0.6B, SkillRet-Embedding-8B, Qwen3-Embedding-8B, harrier-oss-v1-0.6b). Ranks 21–60 from each retriever are merged into one pool of non-relevant candidates, and 15 negatives are sampled per positive.
• Hardware: 4× NVIDIA B200 GPUs (DDP)
• Effective batch size: 384 (96 per device × 4 GPUs)
• Max sequence length: 8,192 tokens
• Learning rate: 2e-5, warmup ratio 0.1
• Schedule: one epoch
• Precision: BF16

Evaluation Results

Evaluated on the SkillRet benchmark evaluation split (4,392 queries, 6,006 skills). The reranker rescores the top-20 candidates returned by SkillRet-Embedding-8B.

Full metrics for this model:

Intended Use

This model is designed to rerank candidate agent skills for a natural-language user request. It is part of the SkillRet benchmark submission for evaluating skill retrieval systems for AI agents.

Limitations

• Optimized for English-language queries and agent skills.
• Reranking is substantially slower than first-stage retrieval. Rescoring 20 candidates takes about 0.8 s per query on one B200, so latency-sensitive deployments may use the embedding model alone.

Citation

Citation information will be added in the de-anonymized release.