DeepHat-V1-7B-Mixed-IQ4

Aronstk15/DeepHat-V1-7B-Mixed-IQ4

This is a custom mixed-precision quantization of DeepHat/DeepHat-V1-7B.

🧠 Quantization Details

• Base Model: DeepHat-V1-7B
• Attention & LM Head Layers: Kept at FP16 to preserve reasoning, routing, and coding accuracy.
• Feed-Forward (FFN) Layers: Quantized to IQ4_XS (4-bit non-linear) using an importance matrix (imatrix.dat) calibrated on code and technical text.
• Tool Used: llama.cpp (llama-imatrix and llama-quantize)

💡 Why Mixed Precision?

By keeping the highly sensitive Attention and Output layers in FP16, this model retains nearly 100% of the original model's intelligence while significantly reducing the file size and VRAM footprint.

🚀 How to use

You can run this model using llama.cpp, Colibrí, or any GGUF-compatible inference engine.

# Example using llama.cpp / Colibrí
./llama-server -m DeepHat-Mixed-IQ4.gguf -ngl 99 -c 8192