Llama-3.2-3B-Instruct-OpenVINO-INT4

Llama-3.2-3B-Instruct-OpenVINO-INT4 (Disruption Series)

This repository contains an optimized OpenVINO™ IR version of Meta's Llama-3.2-3B-Instruct, quantized to INT4 precision via NNCF. It is engineered for high-performance, low-latency local inference on Windows-based AI workstations and edge devices.

---

🐍 Python Inference (Optimum-Intel)

To run this 3B engine locally:

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "CelesteImperia/Llama-3.2-3B-Instruct-OpenVINO-INT4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)

prompt = "Analyze the technical advantages of INT4 quantization on Intel hardware."
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

💻 For C# / .NET Users (LLamaSharp Implementation)

using LLama.Common;
using LLama;

// 1. Initialize the OpenVINO Model for Llama 3.2
var parameters = new ModelParams("path/to/openvino_model.xml")
{
ContextSize = 4096,
GpuLayerCount = 0 // Optimized for Intel CPU/iGPU/NPU via OpenVINO
};

// 2. Load Weights and Create Context
using var weights = LLamaWeights.LoadFromFile(parameters);
using var context = weights.CreateContext(parameters);
var executor = new StatelessExecutor(weights, parameters);

// 3. Run Inference
var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.User, "Analyze the benefits of running Llama 3.2 locally.");

foreach (var text in executor.InferAsync(chatHistory, new InferenceParams { MaxTokens = 256 }))
{
Console.Write(text);
}

---

🏗️ Technical Forge

• Quantization: INT4 Asymmetric (NNCF)
• Workstation: Dual-GPU (RTX 3090 + RTX A4000)
• Inference Engine: OpenVINO™ Runtime 2026.0

Connect with the architect: Abhishek Jaiswal on LinkedIn

---

☕ Support the Forge

Maintaining a dual-GPU AI workstation and hosting high-bandwidth models requires significant resources. If our open-source tools power your projects, consider supporting our development:

Scan to support via UPI (India Only):

<img src="https://huggingface.co/datasets/CelesteImperia/Assets/resolve/main/QrCode.jpeg" width="200">