flan-t5-xxl-encoder-only-GGUF

InsecureErasure/flan-t5-xxl-encoder-only-GGUF

Quantized GGUF version of Google FLAN T5-XXL (encoder only)

This repo contains additional GGUF quantizations of Google's FLAN-T5 XXL (encoder only) not available in silveroxides/flan-t5-xxl-encoder-only-GGUF, produced using llama.cpp.

These are aimed at users with limited hardware — requiring less memory and storage than higher quantization formats. Other repositories on HF include the decoder layers, resulting in a much bigger file. If you're using this for generating embeddings for Chroma or FLUX models (as I am), you only need the encoder layers.

Big kudos to silveroxides.

<table style="border-collapse: collapse;"> <tr style="border: 1px;"> <td style="writing-mode: vertical-lr; transform: rotate(180deg); text-align: center; border: 1px;"> <strong>Chroma Flash Q5_K_S</strong> </td> <td align="center" style="border: 1px;"><a href="https://huggingface.co/InsecureErasure/flan-t5-xxl-encoder-only-GGUF/blob/main/images/chroma-flan-t5-xxl-Q4_K_M.png"><img src="images/chroma-flan-t5-xxl-Q4_K_M.png" width="350"/></a></td> <td align="center" style="border: 1px;"><a href="https://huggingface.co/InsecureErasure/flan-t5-xxl-encoder-only-GGUF/blob/main/images/chroma-flan-t5-xxl-Q3_K_M.png"><img src="images/chroma-flan-t5-xxl-Q3_K_M.png" width="350"/></a></td> </tr> <tr style="border: 1px;"> <td style="writing-mode: vertical-lr; transform: rotate(180deg); text-align: center; border: 1px;"> <strong>Chroma Flash Q4_K_S</strong> </td> <td align="center" style="border: 1px;"><a href="https://huggingface.co/InsecureErasure/flan-t5-xxl-encoder-only-GGUF/blob/main/images/chroma-Q4_flan-t5-xxl-Q4.png"><img src="images/chroma-Q4_flan-t5-xxl-Q4.png" width="350"/></a></td> <td align="center" style="border: 1px;"><a href="https://huggingface.co/InsecureErasure/flan-t5-xxl-encoder-only-GGUF/blob/main/images/chroma-Q4_flan-t5-xxl-Q3.png"><img src="images/chroma-Q4_flan-t5-xxl-Q3.png" width="350"/></a></td> </tr> <tr style="border: 1px;"> <td style="border: 1px;"></td> <td align="center" style="border: 1px;"><strong>flan-t5-xxl-TE-only Q4_K_M</strong></td> <td align="center" style="border: 1px;"><strong>flan-t5-xxl-TE-only Q3_K_M</strong></td> </tr> </table>