octo-planner-gguf

Quantized Octo-planner: On-device Language Model for Planner-Action Agents Framework

This repo includes GGUF quantized models, for our Octo-planner model at NexaAIDev/octopus-planning

GGUF Quantization

To run the models, please download them to your local machine using either git clone or Hugging Face Hub

git clone https://huggingface.co/NexaAIDev/octo-planner-gguf

Run with llama.cpp (Recommended)

1. Clone and compile:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# Compile the source code:
make

1. Execute the Model:

Run the following command in the terminal:

./llama-cli -m ./path/to/octopus-planning-Q4_K_M.gguf -p "<|user|>Find my presentation for tomorrow's meeting, connect to the conference room projector via Bluetooth, increase the screen brightness, take a screenshot of the final summary slide, and email it to all participants<|end|><|assistant|>"

Run with Ollama

Since our models have not been uploaded to the Ollama server, please download the models and manually import them into Ollama by following these steps:

1. Install Ollama on your local machine. You can also following the guide from Ollama GitHub repository

git clone https://github.com/ollama/ollama.git ollama

1. Locate the local Ollama directory:

cd ollama

1. Create a Modelfile in your directory

touch Modelfile

1. In the Modelfile, include a FROM statement with the path to your local model, and the default parameters:

FROM ./path/to/octopus-planning-Q4_K_M.gguf

1. Use the following command to add the model to Ollama:

ollama create octopus-planning-Q4_K_M -f Modelfile

1. Verify that the model has been successfully imported:

ollama ls

1. Run the mode

ollama run octopus-planning-Q4_K_M "<|user|>Find my presentation for tomorrow's meeting, connect to the conference room projector via Bluetooth, increase the screen brightness, take a screenshot of the final summary slide, and email it to all participants<|end|><|assistant|>"

Quantized GGUF Models Benchmark

Quantized with llama.cpp