Downloads · 30 days
319
9% of all-time downloads
glogwa68/granite-4.0-h-350m-DISTILL-gemini-5M-CTX-GGUF
granite-4.0-h-350m-DISTILL-gemini-5M-CTX-GGUF is a machine learning model from glogwa68. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This model has been modified to support 5 million token context natively.
Downloads · 30 days
319
9% of all-time downloads
All-time downloads
3.7K
Public
Repo size
4 GB
Likes
2
Public
Click a slice to open those files.
.gguf4 GB · 100%
From the Hugging Face model README
This model has been modified to support 5 million token context natively.
from llama_cpp import Llama
llm = Llama(
model_path="./granite-4.0-h-350m-DISTILL-gemini-3-pro-think-f16-5M-ctx.gguf",
n_ctx=5242880, # 5M context
n_gpu_layers=-1,
tensor_split=[1,1,1,1,1,1,1,1], # 8 GPUs
)
output = llm("Your prompt here", max_tokens=500)
| File | Quantization | Size |
|---|---|---|
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-f16-5M-ctx.gguf | See name | 0.64 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q2_k-5M-ctx.gguf | See name | 0.15 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q3_k_l-5M-ctx.gguf | See name | 0.18 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q3_k_m-5M-ctx.gguf | See name | 0.18 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q3_k_s-5M-ctx.gguf | See name | 0.17 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q4_0-5M-ctx.gguf | See name | 0.20 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q4_1-5M-ctx.gguf | See name | 0.22 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q4_k_m-5M-ctx.gguf | See name | 0.21 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q4_k_s-5M-ctx.gguf | See name | 0.20 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q5_0-5M-ctx.gguf | See name | 0.23 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q5_1-5M-ctx.gguf | See name | 0.25 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q5_k_m-5M-ctx.gguf | See name | 0.24 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q5_k_s-5M-ctx.gguf | See name | 0.23 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q6_k-5M-ctx.gguf | See name | 0.26 GB |
| granite-4.0-h-350m-DISTILL-gemini-3-pro-think-q8_0-5M-ctx.gguf | See name | 0.34 GB |