granite-4.0-h-tiny-gguf

granite-4.0-h-tiny-gguf

🚀 GGUF-quantized release of IBM Granite 4.0 H Tiny, converted for efficient local inference.

This repository provides GGUF-format versions of the original
ibm-granite/granite-4.0-h-tiny model, enabling fast execution with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

---

Model Overview

• Base Model: IBM Granite 4.0 H Tiny
• Model Type: Decoder-only Transformer
• Format: GGUF
• Pipeline: Text Generation
• Language: English

Designed for lightweight, low-latency inference while preserving strong instruction-following capabilities.

---

What This Repository Contains

• ✅ GGUF-converted model weights
• ✅ Quantized for CPU / low-VRAM environments
• ✅ Ready for local & edge inference
• ❌ No fine-tuning or weight modification (format conversion only)

---

Usage

Ollama

Create a Modelfile:

FROM granite-4.0-h-tiny.gguf

Run:

ollama create granite-tiny -f Modelfile
ollama run granite-tiny

---

LM Studio / GUI Tools

1. Open LM Studio
2. Load the .gguf file
3. Select a llama.cpp backend
4. Start inference

---

Quantization Notes

GGUF quantization provides:

• Reduced memory usage
• Faster load times
• Compatibility with CPU-only systems

Recommended usage:

• Q4 / Q5: Best balance of speed & quality
• Q8: Higher quality, more memory

---

Intended Use Cases

• Text generation
• Review & sentiment analysis
• QA automation pipelines
• Agentic systems (RAG, MCP, LangGraph, CrewAI)
• Offline / embedded AI applications

---

Attribution

• Original Model:
  IBM Granite Team
  https://huggingface.co/ibm-granite/granite-4.0-h-tiny

• GGUF Conversion & Packaging:
  Kavindu Hansaka Jayasinghe

---

License

This repository follows the MIT license, consistent with the original model.

Please review the base model license before commercial usage.

---

Disclaimer

This is an unofficial GGUF conversion.
IBM is not affiliated with or responsible for this release.