Downloads · 30 days
34
18% of all-time downloads
Disya/Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf
Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf is a machine learning model from Disya. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Downloads · 30 days
34
18% of all-time downloads
All-time downloads
187
Public
Repo size
16.6 GB
Likes
1
Public
Click a slice to open those files.
.gguf16.6 GB · 100%
From the Hugging Face model README
I tried creating an imatrix on a calibration dataset for RP, so performance outside of RP might be slightly worse compared to regular IQ4_XS quants. Also, I kept the embeddings in better quality.
8 bit Output tensor/embed (That added about 200MB)
Install llama.cpp through brew (works on Mac and Linux)
brew install llama.cpp
Invoke the llama.cpp server or the CLI.
build/bin/llama-cli --hf-repo Disya/Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf --hf-file Pantheon-Proto-RP-1.8-30B-A3B-IQ4_XS.gguf -p "The meaning to life and the universe is"
build/bin/llama-server --hf-repo Disya/Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf --hf-file Pantheon-Proto-RP-1.8-30B-A3B-IQ4_XS.gguf -c 2048
Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
git clone https://github.com/ggml-org/llama.cpp
Step 2: Move into the llama.cpp folder and build it with -DLLAMA_CURL=ON flag along with other hardware-specific flags (for ex: -DGGML_CUDA=ON for Nvidia GPUs on Linux).
cd llama.cpp && cmake -B build -DLLAMA_CURL=ON && cmake --build build --config Release -j
Step 3: Run inference through the main binary.
build/bin/llama-cli --hf-repo Disya/Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf --hf-file Pantheon-Proto-RP-1.8-30B-A3B-IQ4_XS.gguf -p "The meaning to life and the universe is"
or
build/bin/llama-server --hf-repo Disya/Gryphe-Pantheon-Proto-RP-1.8-30B-A3B-Q4_XS-gguf --hf-file Pantheon-Proto-RP-1.8-30B-A3B-IQ4_XS.gguf -c 2048