octo-net-gguf

Octopus V4-GGUF: Graph of language models

<p align="center"> - <a href="https://huggingface.co/NexaAIDev/Octopus-v4" target="_blank">Original Model</a> - <a href="https://www.nexa4ai.com/" target="_blank">Nexa AI Website</a> - <a href="https://github.com/NexaAI/octopus-v4" target="_blank">Octopus-v4 Github</a> - <a href="https://arxiv.org/abs/2404.19296" target="_blank">ArXiv</a> - <a href="https://huggingface.co/spaces/NexaAIDev/domain_llm_leaderboard" target="_blank">Domain LLM Leaderbaord</a> </p> <p align="center" width="100%"> <a><img src="octopus-v4-logo.png" alt="nexa-octopus" style="width: 40%; min-width: 300px; display: block; margin: auto;"></a> </p>

Acknowledgement:
We sincerely thank our community members, Mingyuan and Zoey, for their extraordinary contributions to this quantization effort. Please explore Octopus-v4 for our original huggingface model.

Get Started

To run the models, please download them to your local machine using either git clone or Hugging Face Hub

git clone https://huggingface.co/NexaAIDev/octopus-v4-gguf

Run with llama.cpp (Recommended)

1. Clone and compile:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# Compile the source code:
make

1. Execute the Model:

Run the following command in the terminal:

./main -m ./path/to/octopus-v4-Q4_K_M.gguf -n 256 -p "<|system|>You are a router. Below is the query from the users, please call the correct function and generate the parameters to call the function.<|end|><|user|>Tell me the result of derivative of x^3 when x is 2?<|end|><|assistant|>"

Run with Ollama

Since our models have not been uploaded to the Ollama server, please download the models and manually import them into Ollama by following these steps:

1. Install Ollama on your local machine. You can also following the guide from Ollama GitHub repository

git clone https://github.com/ollama/ollama.git ollama

1. Locate the local Ollama directory:

cd ollama

1. Create a Modelfile in your directory

touch Modelfile

1. In the Modelfile, include a FROM statement with the path to your local model, and the default parameters:

FROM ./path/to/octopus-v4-Q4_K_M.gguf
PARAMETER temperature 0
PARAMETER num_ctx 1024
PARAMETER stop <nexa_end>

1. Use the following command to add the model to Ollama:

ollama create octopus-v4-Q4_K_M -f Modelfile

1. Verify that the model has been successfully imported:

ollama ls

1. Run the model

ollama run octopus-v4-Q4_K_M "<|system|>You are a router. Below is the query from the users, please call the correct function and generate the parameters to call the function.<|end|><|user|>Tell me the result of derivative of x^3 when x is 2?<|end|><|assistant|>"

Dataset and Benchmark

• Utilized questions from MMLU to evaluate the performances.
• Evaluated with the Ollama llm-benchmark method.

Quantized GGUF Models

Quantized with llama.cpp