Downloads · 30 days
129
7% of all-time downloads
himalaya-ai/himalayagpt-0.5b-it-gguf
himalayagpt-0.5b-it-gguf is a text generation model from himalaya-ai. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GGUF quantizations of himalaya-ai/himalayagpt-0.5b-it by the Himalaya AI team.
Downloads · 30 days
129
7% of all-time downloads
All-time downloads
1.8K
Public
Repo size
3.8 GB
Likes
6
Public
Click a slice to open those files.
.gguf1.9 GB · 100%
From the Hugging Face model README
GGUF quantizations of himalaya-ai/himalayagpt-0.5b-it by the Himalaya AI team.
For everything about the model itself — architecture, training, intended use, limitations, languages — see the base model card. This page only documents the quantization.
These GGUFs do not run on upstream ggml-org/llama.cpp. Nanochat is not a supported architecture upstream — vanilla llama.cpp will reject the file with Model NanochatForCausalLM is not supported.
Use the fork: HimalayaAI/llama.cpp.
LM Studio, Ollama, Open WebUI, and other UIs that bundle vanilla llama.cpp will not work until upstream ships nanochat support.
| File | Size | Notes |
|---|---|---|
himalayagpt-0.5b-it-bf16.gguf | 1.0 GB | Recommended. Numerically equivalent to the HF Transformers reference. |
himalayagpt-0.5b-it-Q8_0.gguf | 533 MB | Drop-in for BF16; near-identical quality. |
himalayagpt-0.5b-it-Q4_K_M.gguf | 300 MB | Smallest. Math accuracy notably worse than BF16/Q8_0. |
Don't build an F16 GGUF. The model's squared-ReLU MLP overflows F16's ±65504 range and yields NaN logits. BF16 has the same on-disk size with F32 dynamic range — use that instead.
The chat template (<|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|>) and EOT token are embedded in the GGUF metadata.
OpenAI-style system messages are supported and merge into the next user turn (separated by a blank line). Tool/developer roles render as user content. The embedded template also strips the 9 nanochat control-token literal strings from message content to prevent prompt injection — if you want a literal <|assistant_start|> in user input, that will not survive into the rendered prompt by design.
Build the fork, then run llama-cli:
git clone --recursive https://github.com/HimalayaAI/llama.cpp.git
cd llama.cpp
cmake -S . -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_TOOLS=ON
cmake --build build --target llama-cli -j$(nproc)
./build/bin/llama-cli \
-m himalayagpt-0.5b-it-Q8_0.gguf \
--jinja --temp 0.2 --top-k 40 --repeat-penalty 1.08 -n 256
Recommended sampling (matches the upstream Himalaya AI Colab): temperature=0.2, top_k=40, repetition_penalty=1.08. Greedy (temp=0) loops on long answers.
Full build/deploy notes: https://github.com/lukas-h/selfhosted-himalaya-gpt.
Verified against the upstream Hugging Face Transformers implementation: