rank-zephyr-7b-v1-full-GGUF

RankZephyr-7B-v1-Full GGUF

GGUF quantizations of castorini/rank_zephyr_7b_v1_full.

Quantization Variants

VRAM Requirements

Usage

llama.cpp

./main -m model_q5_k_m.gguf -n 256 --chat-template mistral -p "Your prompt here"

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="model_q5_k_m.gguf",
    n_gpu_layers=-1,
    n_ctx=2048,
    chat_format="mistral"
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Your prompt here"}],
    max_tokens=256
)
print(response["choices"][0]["message"]["content"])

LM Studio

1. Download any .gguf file from this repo
2. Open in LM Studio
3. Configure context and parameters
4. Start chat

Ollama

ollama create rankzephyr -f Modelfile

Modelfile:

FROM model_q5_k_m.gguf
PARAMETER temperature 0.7

Base Model

castorini/rank_zephyr_7b_v1_full — Fine-tuned version of Zephyr-7B-β designed for listwise reranking tasks. State-of-the-art open-source reranking model achieving 0.7803 NDCG@10 on DL19 and 0.8211 on DL20 with SPLADE++ ED first-stage retrieval.

Performance

• DL19 NDCG@10: 0.7803
• DL20 NDCG@10: 0.8211
• Trained on: RankGPT-4 reorderings of OpenAI Ada2 embeddings
• Specialized for: Zero-shot listwise reranking

Use Case

RankZephyr is designed to rerank search results produced by first-stage retrievers. Provide it with a query and a list of documents, and it will output a reordered ranking based on relevance.

License

MIT (as per base model)

Citation

@ARTICLE{pradeep2023rankzephyr,
  title   = {{RankZephyr}: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!},
  author  = {Ronak Pradeep and Sahel Sharifymoghaddam and Jimmy Lin},
  year    = {2023},
  journal = {arXiv:2312.02724}
}