mistral-orpo-capybara-7k-GGUF

Mistral-ORPO-Capybara-7k GGUF

GGUF quantizations of kaist-ai/mistral-orpo-capybara-7k.

Quantization Variants

VRAM Requirements

Usage

llama.cpp

./main -m model_q5_k_m.gguf -n 256 --chat-template mistral -p "Your prompt here"

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="model_q5_k_m.gguf",
    n_gpu_layers=-1,
    n_ctx=2048,
    chat_format="mistral"
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Your prompt here"}],
    max_tokens=256
)
print(response["choices"][0]["message"]["content"])

LM Studio

1. Download any .gguf file from this repo
2. Open in LM Studio
3. Configure context and parameters
4. Start chat

Ollama

ollama create mistral-orpo -f Modelfile

Modelfile:

FROM model_q5_k_m.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Base Model

kaist-ai/mistral-orpo-capybara-7k — Fine-tuned version of Mistral-7B-v0.1 using ORPO (Odds Ratio Preference Optimization) on the Capybara DPO dataset. Trained for 2.5 hours on four A100s.

Performance

• AlpacaEval 2.0 (LC): 15.9%
• MT-Bench: 7.44
• IFEval Inst-Strict: 58.27%

License

MIT (as per base model)

Citation

@misc{hong2024orpo,
      title={ORPO: Monolithic Preference Optimization without Reference Model},
      author={Jiwoo Hong and Noah Lee and James Thorne},
      year={2024},
      eprint={2403.07691},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}