Phi-4-mini-instruct-GGUF

Phi-4-mini-instruct GGUF

GGUF versions of Microsoft Phi-4-mini-instruct prepared for efficient local inference.

This repository contains an F16 GGUF version and a Q4_K_M quantized version.

Available Files

Recommended Version

Phi-4-mini-instruct-Q4_K_M.gguf is recommended for most consumer hardware and local inference setups.

It provides a substantially smaller memory footprint than F16 while retaining the advantages of the GGUF format.

Original Model

The original model is Microsoft's official Phi-4-mini-instruct.

Original model:

https://huggingface.co/microsoft/Phi-4-mini-instruct

Please refer to Microsoft's original model repository for technical details, intended use, limitations, and licensing.

Quantization

The Q4_K_M file was generated from the F16 GGUF model using llama.cpp quantization tools.

Quantization format:

Q4_K_M

llama.cpp

Basic usage:

llama-cli -m Phi-4-mini-instruct-Q4_K_M.gguf

GPU offloading:

llama-cli -m Phi-4-mini-instruct-Q4_K_M.gguf -ngl 99

Local server:

llama-server -m Phi-4-mini-instruct-Q4_K_M.gguf -ngl 99

Compatible Software

• llama.cpp
• llama-cpp-python
• LM Studio
• Jan
• Other GGUF-compatible runtimes

Model Variants

F16

Higher precision with substantially larger memory requirements.

Q4_K_M

Smaller model size and lower memory requirements, making it more suitable for local deployment on consumer hardware.

Disclaimer

This is a community GGUF conversion and quantization. It is not an official Microsoft repository.

License

Please follow the licensing terms of the original Microsoft Phi-4-mini-instruct model.

SHA256

Checksums can be generated with:

sha256sum *.gguf