Llama-3.2-1B-Instruct-4bit-MLX

⚡️ Llama-3.2-1B-Instruct (4-Bit MLX)

This repository hosts a 4-bit quantized artifact of Meta's Llama-3.2-1B-Instruct foundation model, engineered specifically for high-throughput, low-latency on-device inference on Apple Silicon using Apple's open-source MLX framework.

---

📊 Hardware Benchmarks & Performance Telemetry

Telemetry was recorded running native INT4 execution passes across unified memory on Apple Silicon:

🧠 KV Cache Memory Footprint (Grouped-Query Attention)

Thanks to Grouped-Query Attention (GQA with 8 KV heads vs. 32 query heads), memory allocation scales minimally over extended context lengths:

• Theoretical Cache Growth: $$2 \times 16\text{ layers} \times 8\text{ KV heads} \times 64\text{ dim} \times 1200\text{ tokens} \times 2\text{ bytes} \approx \mathbf{37.5\text{ MB}}$$
• Edge Safety: The combined INT4 weights + KV cache footprint remains under 0.8 GB, ensuring safe execution on constrained iOS runtimes without Jetsam memory termination risks.

---

📈 Latency Dynamics: Cold Start vs. Warm Start Prefill

A primary bottleneck in on-device AI infrastructure is the latency disparity between uninitialized storage reads and active Unified Memory caching:

1. Cold Start Prefill (44.64 tok/s) — High I/O Storage Penalty
┌─────────────┐   Disk Read & Bus Transfer   ┌────────────────┐   Compute Pass   ┌──────────────┐
│ NVMe / SSD  │ ───────────────────────────► │ Unified Memory │ ───────────────► │ GPU Compute  │
└─────────────┘           (~22.4 ms/tok)     └────────────────┘                  └──────────────┘

2. Warm Start Prefill (778.79 tok/s) — Zero Storage Latency (~17.4x Acceleration)
                                             ┌────────────────┐   Direct Stream  ┌──────────────┐
                                             │ Resident Cache │ ───────────────► │ GPU Compute  │
                                             └────────────────┘   (~1.28 ms/tok) └──────────────┘

Prefill Latency Comparison (Tokens per Second)
──────────────────────────────────────────────────────────────────────────────
Cold Start (Disk I/O)    | ██ (44.64 tok/s)
Warm Start (RAM Cached)  | ████████████████████████████████████ (778.79 tok/s) [17.4x]
──────────────────────────────────────────────────────────────────────────────

---

🚀 Quickstart & Inference

1. Installation

Ensure an Apple Silicon Mac with macOS 14+ and an isolated virtual environment:

pip install mlx-lm huggingface_hub

2. Direct CLI Generation

Run text generation directly from your terminal using the MLX engine:

mlx_lm.generate \
    --model rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX \
    --prompt "Explain how 4-bit quantization reduces memory pressure on Apple Silicon." \
    --max-tokens 300

3. Native Python Pipeline

from mlx_lm import load, generate

# Load the 4-bit model directly from Hugging Face Hub
model, tokenizer = load("rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX")

prompt = "Explain how 4-bit quantization reduces memory pressure on Apple Silicon."

response = generate(
    model, 
    tokenizer, 
    prompt=prompt, 
    max-tokens=300, 
    verbose=True
)

print(response)

---

🛠️ Quantization Pipeline & Methodology

Converted from official meta-llama/Llama-3.2-1B-Instruct bfloat16 tensors using mlx_lm.convert:

# 1. Force snapshot metadata caching
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('meta-llama/Llama-3.2-1B-Instruct')"

# 2. Execute 4-bit quantization
mlx_lm.convert \
    --hf-path meta-llama/Llama-3.2-1B-Instruct \
    -q \
    --q-bits 4 \
    --upload-repo rajib-sarwar/Llama-3.2-1B-Instruct-4bit-MLX

---

🛡️ Edge Safety & On-Device Memory Design

• Jetsam Compliance: Operating at < 830 MB peak memory overhead ensures mobile applications running on iOS/iPadOS can load and stream responses without risking operating system Jetsam Out-of-Memory (OOM) termination crashes.
• Unified Memory Bus Optimization: Quantizing weights to 4-bit cuts required memory bandwidth by ~70%, minimizing battery drain and preventing thermal throttling on constrained devices.

---

📄 License & Attribution

This model is a quantized derivative of Meta's Llama 3.2 family. Usage is subject to the Meta Llama 3.2 Community License Agreement.