h2o-danube3-4b-chat-GGUF

h2o-danube3-4b-chat-GGUF

• Model creator: H2O.ai
• Original model: h2oai/h2o-danube3-4b-chat

Description

This repo contains GGUF format model files for h2o-danube3-4b-chat quantized using llama.cpp framework.

Table below summarizes different quantized versions of h2o-danube3-4b-chat. It shows the trade-off between size, speed and quality of the models.

Columns in the table are:

• Name -- model name and link
• Quant method -- quantization method
• Model size -- size of the model in gigabytes
• MT-Bench AVG -- MT-Bench benchmark score. The score is from 1 to 10, the higher, the better
• Perplexity -- perplexity metric on WikiText-2 dataset. It's reported in a perplexity test from llama.cpp. The lower, the better
• Tokens per second -- generation speed in tokens per second, as reported in a perplexity test from llama.cpp. The higher, the better. Speed tests are done on a single H100 GPU

Prompt template

<|prompt|>Why is drinking water so healthy?</s><|answer|>