skillcorpus-reranker-0.6b

skillcorpus-reranker-0.6b

A cross-encoder for agent-skill retrieval: given a task and a candidate skill document, judge whether the skill helps. Fine-tuned from Qwen/Qwen3-Reranker-0.6B.

Reranks the candidates recalled by skillcorpus-embedding-0.6b. Because scoring is one forward pass per (task, skill) pair, run it on a shortlist — typically the encoder's top 20–50 — not the whole corpus. The skill documents it was built to score have the schema of skillcorpus-demo-1k.

Requires transformers>=4.56 (the dtype= argument was named torch_dtype= before that).

Usage

The model answers a yes/no question; the relevance score is the softmax over the yes and no logits at the final position.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "EverMind-AI/skillcorpus-reranker-0.6b"
tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16).cuda().eval()

PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
          'based on the Query and the Instruct provided. Note that the answer can '
          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'

YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")


def score(pairs, max_length=4096):
    prompts = [PREFIX + p + SUFFIX for p in pairs]
    enc = tok(prompts, padding=True, truncation=True,
              max_length=max_length, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**enc).logits[:, -1, :]
    pair = torch.stack([logits[:, NO], logits[:, YES]], dim=-1)
    return torch.softmax(pair, dim=-1)[:, 1].float().tolist()


INSTRUCT = ("Given a task description, judge whether the skill document "
            "is relevant and useful for completing the task")


def pair(task, name, description, body):
    return (f"<Instruct>: {INSTRUCT}\n\n"
            f"<Query>: {task}\n\n"
            f"<Document>: {name} | {description} | {body}")


task = "resolve conflicts after a git merge"
print(score([
    pair(task, "resolve-conflicts", "Resolve git merge conflicts.", "..."),
    pair(task, "sourdough", "Bake sourdough bread.", "..."),
]))
# -> [0.97, 0.01]

Exact scores shift in the last decimal with dtype and hardware; the ordering is what matters.

Two things to keep intact: the PREFIX / SUFFIX template, since the score is read off the final-position logits, and the <Instruct> / <Query> / <Document> layout with blank lines between the parts. Truncate the document body, not the template.

Via the chat template

chat_template.jinja reproduces exactly the string built above, so you can let the tokenizer assemble it instead. It reads three roles — system carries the instruction (omit it to get the default shown above), query the task, and document the skill:

prompt = tok.apply_chat_template([
    {"role": "system",   "content": INSTRUCT},
    {"role": "query",    "content": task},
    {"role": "document", "content": "resolve-conflicts | Resolve git merge conflicts. | ..."},
], tokenize=False)

Truncation used in training

Beyond the token-level max_length, each field was cut to a fixed number of characters before the prompt was assembled. Matching this keeps inference inputs on the same distribution as training:

Intended use

Second-stage reranking over a shortlist recalled by skillcorpus-embedding-0.6b — typically its top 20–50. Scoring is one forward pass per (task, skill) pair, so it does not scale to a whole registry. Treat the output as a ranking signal rather than a calibrated probability: compare scores within one candidate list, not across different tasks.

Citation

@article{wang2026skillcorpus,
  title         = {SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents},
  author        = {Wang, Yanze and Yao, Pengfei and Sun, Tianyi and Hu, Chuanrui and Xiao, Yan and Luo, Xiaotian and Han, Yunyun and Chen, Yifan and Sun, Jun and Deng, Yafeng},
  year          = {2026},
  eprint        = {2607.15557},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2607.15557}
}

License

Apache-2.0, inherited from the base model. Skills in the corpus keep their own upstream licenses.