Phi-3-medium-128k-instruct-GGUF

Phi-3 Medium 128k Instruct (GGUF)

This repository provides GGUF quantized models for
Microsoft Phi-3 Medium 128k Instruct.

Quantization was performed using llama.cpp with Importance Matrix (IQ) techniques to optimize VRAM usage, inference speed, and model quality.

---

📊 Benchmarks (NVIDIA T4)

Benchmarks were collected on Google Colab (Tesla T4) using llama-bench and llama-perplexity.

Notes

• IQ4_XS delivers ~87% higher throughput than Q5_K_M with negligible quality loss.
• 2-bit quantization causes severe degradation on Phi-3 Medium.

---

📦 Available Files

RAM estimates include KV-cache overhead.

---

💡 Quantization Info

Importance Matrix (IQ) quantization uses calibration data to preserve the most important weights, significantly reducing quality loss compared to standard K-quant methods at similar sizes.

Recommended: IQ4_XS
Fits in ~8GB VRAM and sustains 20+ tokens/sec on consumer GPUs (RTX 3060 / 3070 / 4060).

---

🚀 Usage

⚠️ Choose ONE of the following options. Do NOT run both.

---

Option 1: CLI (llama.cpp)

Use this if you want an interactive terminal session.

./llama-cli \
  -m Phi-3-medium-128k-instruct-IQ4_XS.gguf \
  -n -1 \
  --color \
  -cnv

Option 2: Python (llama-cpp-python)

from llama_cpp import Llama

# Set gpu_layers to -1 to offload all layers to GPU
llm = Llama(
   model_path="./Phi-3-medium-128k-instruct-IQ4_XS.gguf",
   n_ctx=4096, 
   n_gpu_layers=-1,
   verbose=True
)

output = llm.create_chat_completion(
   messages=[
       {"role": "system", "content": "You are a helpful assistant."},
       {"role": "user", "content": "Explain quantum physics."},
   ]
)

print(output['choices'][0]['message']['content'])