Downloads · 30 days
311
1% of all-time downloads
Mungert/kanana-1.5-8b-instruct-2505-GGUF
kanana-1.5-8b-instruct-2505-GGUF is a text generation model from Mungert. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model was generated using llama.cpp at commit f5cd27b7.
Downloads · 30 days
311
1% of all-time downloads
All-time downloads
26.3K
Public
Repo size
145 GB
Likes
3
Public
Click a slice to open those files.
.gguf145 GB · 100%
From the Hugging Face model README
This model was generated using llama.cpp at commit f5cd27b7.
Our latest quantization method introduces precision-adaptive quantization for ultra-low-bit models (1-2 bit), with benchmark-proven improvements on Llama-3-8B. This approach uses layer-specific strategies to preserve accuracy while maintaining extreme memory efficiency.
All tests conducted on Llama-3-8B-Instruct using:
| Quantization | Standard PPL | DynamicGate PPL | Δ PPL | Std Size | DG Size | Δ Size | Std Speed | DG Speed |
|---|---|---|---|---|---|---|---|---|
| IQ2_XXS | 11.30 | 9.84 | -12.9% | 2.5G | 2.6G | +0.1G | 234s | 246s |
| IQ2_XS | 11.72 | 11.63 | -0.8% | 2.7G | 2.8G | +0.1G | 242s | 246s |
| IQ2_S | 14.31 | 9.02 | -36.9% | 2.7G | 2.9G | +0.2G | 238s | 244s |
| IQ1_M | 27.46 | 15.41 | -43.9% | 2.2G | 2.5G | +0.3G | 206s | 212s |
| IQ1_S | 53.07 | 32.00 | -39.7% | 2.1G | 2.4G | +0.3G | 184s | 209s |
Key:
Key Improvements:
Tradeoffs:
📌 Fitting models into GPU VRAM
✔ Memory-constrained deployments
✔ Cpu and Edge Devices where 1-2bit errors can be tolerated
✔ Research into ultra-low-bit quantization
Selecting the correct model format depends on your hardware capabilities and memory constraints.
📌 Use BF16 if:
✔ Your hardware has native BF16 support (e.g., newer GPUs, TPUs).
✔ You want higher precision while saving memory.
✔ You plan to requantize the model into another format.
📌 Avoid BF16 if:
❌ Your hardware does not support BF16 (it may fall back to FP32 and run slower).
❌ You need compatibility with older devices that lack BF16 optimization.
📌 Use F16 if:
✔ Your hardware supports FP16 but not BF16.
✔ You need a balance between speed, memory usage, and accuracy.
✔ You are running on a GPU or another device optimized for FP16 computations.
📌 Avoid F16 if:
❌ Your device lacks native FP16 support (it may run slower than expected).
❌ You have memory limitations.
Quantization reduces model size and memory usage while maintaining as much accuracy as possible.
📌 Use Quantized Models if:
✔ You are running inference on a CPU and need an optimized model.
✔ Your device has low VRAM and cannot load full-precision models.
✔ You want to reduce memory footprint while keeping reasonable accuracy.
📌 Avoid Quantized Models if:
❌ You need maximum accuracy (full-precision models are better for this).
❌ Your hardware has enough VRAM for higher-precision formats (BF16/F16).
These models are optimized for extreme memory efficiency, making them ideal for low-power devices or large-scale deployments where memory is a critical constraint.
IQ3_XS: Ultra-low-bit quantization (3-bit) with extreme memory efficiency.
IQ3_S: Small block size for maximum memory efficiency.
IQ3_M: Medium block size for better accuracy than IQ3_S.
Q4_K: 4-bit quantization with block-wise optimization for better accuracy.
Q4_0: Pure 4-bit quantization, optimized for ARM devices.
| Model Format | Precision | Memory Usage | Device Requirements | Best Use Case |
|---|---|---|---|---|
| BF16 | Highest | High | BF16-supported GPU/CPUs | High-speed inference with reduced memory |
| F16 | High | High | FP16-supported devices | GPU inference when BF16 isn't available |
| Q4_K | Medium Low | Low | CPU or Low-VRAM devices | Best for memory-constrained environments |
| Q6_K | Medium | Moderate | CPU with more memory | Better accuracy while still being quantized |
| Q8_0 | High | Moderate | CPU or GPU with enough VRAM | Best accuracy among quantized models |
| IQ3_XS | Very Low | Very Low | Ultra-low-memory devices | Extreme memory efficiency and low accuracy |
| Q4_0 | Low | Low | ARM or low-memory devices | llama.cpp can optimize for ARM devices |
kanana-1.5-8b-instruct-2505-bf16.ggufkanana-1.5-8b-instruct-2505-f16.ggufkanana-1.5-8b-instruct-2505-bf16-q8_0.ggufkanana-1.5-8b-instruct-2505-f16-q8_0.ggufkanana-1.5-8b-instruct-2505-q4_k.ggufkanana-1.5-8b-instruct-2505-q4_k_s.ggufkanana-1.5-8b-instruct-2505-q6_k.ggufkanana-1.5-8b-instruct-2505-q8_0.ggufkanana-1.5-8b-instruct-2505-iq3_xs.ggufkanana-1.5-8b-instruct-2505-iq3_m.ggufkanana-1.5-8b-instruct-2505-q4_0.gguf❤ Please click "Like" if you find this useful!
Help me test my AI-Powered Network Monitor Assistant with quantum-ready security checks:
👉 Quantum Network Monitor
💬 How to test:
Choose an AI assistant type:
TurboLLM (GPT-4o-mini)HugLLM (Hugginface Open-source)TestLLM (Experimental CPU-only)I’m pushing the limits of small open-source models for AI network monitoring, specifically:
🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads):
🟢 TurboLLM – Uses gpt-4o-mini for:
🔵 HugLLM – Latest Open-source models:
"Give me info on my websites SSL certificate""Check if my server is using quantum safe encyption for communication""Run a comprehensive security audit on my server"I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.
If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
Thank you! 😊
<p align="center"> <br> <picture> <img src="./assets/logo/kanana-logo.png" width="60%" style="margin: 40px auto;"> </picture> </br> <p align="center"> 🤗 <a href="https://kko.kakao.com/kananallm">1.5 HF Models</a>   |   📕 <a href="https://tech.kakao.com/posts/707">1.5 Blog</a>   |   📜 <a href="https://arxiv.org/abs/2502.18934">Technical Report</a> <br>2025/05/23: Published a blog post about Kanana 1.5 models and released 🤗HF model weights.2025/02/27: Released Technical Report and 🤗HF model weights.2025/01/10: Published a blog post about the development of Kanana Nano model.2024/11/14: Published blog posts (pre-training, post-training) about the development of Kanana models.2024/11/06: Published a presentation video about the development of the Kanana models.Kanana 1.5, a newly introduced version of the Kanana model family, presents substantial enhancements in coding, mathematics, and function calling capabilities over the previous version, enabling broader application to more complex real-world problems. This new version now can handle up to 32K tokens length natively and up to 128K tokens using YaRN, allowing the model to maintain coherence when handling extensive documents or engaging in extended conversations. Furthermore, Kanana 1.5 delivers more natural and accurate conversations through a refined post-training process.
[!Note] Neither the pre-training nor the post-training data includes Kakao user data.
<br>[!Note] * Models released under Apache 2.0 are trained on the latest versions compared to other models.
Currently, the config.json uploaded to HuggingFace is configured for token lengths of 32,768 or less. To process tokens beyond this length, YaRN must be applied. By updating the config.json with the following parameters, you can apply YaRN to handle token sequences up to 128K in length:
"rope_scaling": {
"factor": 4.4,
"original_max_position_embeddings": 32768,
"type": "yarn",
"beta_fast": 64,
"beta_slow": 2
},
<br>
@misc{kananallmteam2025kananacomputeefficientbilinguallanguage,
title={Kanana: Compute-efficient Bilingual Language Models},
author={Kanana LLM Team and Yunju Bak and Hojin Lee and Minho Ryu and Jiyeon Ham and Seungjae Jung and Daniel Wontae Nam and Taegyeong Eo and Donghun Lee and Doohae Jung and Boseop Kim and Nayeon Kim and Jaesun Park and Hyunho Kim and Hyunwoong Ko and Changmin Lee and Kyoung-Woon On and Seulye Baeg and Junrae Cho and Sunghee Jung and Jieun Kang and EungGyun Kim and Eunhwa Kim and Byeongil Ko and Daniel Lee and Minchul Lee and Miok Lee and Shinbok Lee and Gaeun Seo},
year={2025},
eprint={2502.18934},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.18934},
}
<br>