Mistral-7B-Instruct-v0.3-Q4_K_M-GGUF

Mistral-7B-Instruct-v0.3-Q4_K_M GGUF

This repository contains a single GGUF export of mistralai/Mistral-7B-Instruct-v0.3 quantized to Q4_K_M for CPU-oriented llama.cpp usage.

Included Artifact

• File: mistral-7b-Q4_K_M.gguf
• Published size: 4372815936 bytes
• Base model: mistralai/Mistral-7B-Instruct-v0.3
• Quantization: Q4_K_M

This repository does not include the original safetensors weights, the intermediate FP16 GGUF, or additional quantization variants.

Quantization Pipeline

The published file was produced with this workflow:

1. Download the original Hugging Face weights for mistralai/Mistral-7B-Instruct-v0.3.
2. Convert the model to FP16 GGUF with convert_hf_to_gguf.py.
3. Quantize the FP16 GGUF to Q4_K_M with llama-quantize.

The Q4_K_M GGUF in this repository is the runtime artifact intended for normal inference.

TurboQuant

TurboQuant is not baked into the GGUF file stored in this repository.

TurboQuant is a separate runtime optimization that compresses the KV cache to tq3_0. It was validated independently on Windows using the Aaryan-Kapoor/llama.cpp TurboQuant fork with these runtime flags:

--cache-type-k tq3_0 --cache-type-v tq3_0

That means:

• The uploaded model file remains a standard Q4_K_M GGUF.
• TurboQuant changes runtime KV-cache behavior, not the model weights in this repository.
• You only get the TurboQuant memory behavior when running a build that supports tq3_0.

Verified Measurements

The following numbers were measured on Windows CLI runs with n_ctx=4096.

Notes:

• Measured RAM reduction with TurboQuant: 0.39 GB on Windows WorkingSet64.
• Long-context testing was not completed at publication time.
• These results describe the local verification environment and should not be treated as a generic benchmark.

Usage

Standard llama.cpp usage:

llama-server -hf GJ-droid-dev/Mistral-7B-Instruct-v0.3-Q4_K_M-GGUF

Or run locally after downloading the file:

llama-cli \
  -m ./mistral-7b-Q4_K_M.gguf \
  --temp 0 \
  -c 4096

TurboQuant-capable runtime example:

llama-cli \
  -m ./mistral-7b-Q4_K_M.gguf \
  --temp 0 \
  -c 4096 \
  --cache-type-k tq3_0 \
  --cache-type-v tq3_0

Important Notes

• This is a quantized GGUF for llama.cpp-style runtimes.
• TurboQuant support depends on the runtime build and is not embedded in the uploaded file.
• For deterministic extraction or mapping tasks, use temperature=0.