granite-4.2-3b-Heretic-GGUF

Granite 4.2 3B Heretic — GGUF

GGUF quantized versions of tinyopsec/granite-4.2-3b-Heretic — a representation-edited derivative of IBM Granite 4.2 3B.

These files are ready for CPU and GPU inference with llama.cpp, Ollama, LM Studio, Jan, and other GGUF-compatible runtimes.

---

Available Quantizations

If you are unsure which to pick, start with Q5_K_M.

---

Quickstart

llama.cpp

# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh

# Run a chat session directly in the terminal
llama cli -hf tinyopsec/granite-4.2-3b-Heretic-GGUF:Q5_K_M

# Or start an OpenAI-compatible local server
llama serve -hf tinyopsec/granite-4.2-3b-Heretic-GGUF:Q5_K_M

:: Windows (WinGet)
winget install llama.cpp
llama cli -hf tinyopsec/granite-4.2-3b-Heretic-GGUF:Q5_K_M

Manual run with a downloaded file:

./llama-cli \
  -m granite-4.2-3b-Heretic-Q5_K_M.gguf \
  -p "Explain the trade-offs between CPU and GPU inference for LLMs." \
  -n 512

Ollama

ollama run hf.co/tinyopsec/granite-4.2-3b-Heretic-GGUF:Q5_K_M

LM Studio

Open LM Studio → Search → paste tinyopsec/granite-4.2-3b-Heretic-GGUF → download the desired quant.

Docker

docker model run hf.co/tinyopsec/granite-4.2-3b-Heretic-GGUF:Q5_K_M

---

Python — llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="granite-4.2-3b-Heretic-Q5_K_M.gguf",
    n_ctx=8192,
    n_threads=8,
    verbose=False,
)

output = llm(
    "What are the key differences between supervised and reinforcement learning?",
    max_tokens=512,
    temperature=0.7,
    top_p=0.95,
    echo=False,
)

print(output["choices"][0]["text"])

For a chat interface with the model's built-in template:

from llama_cpp import Llama

llm = Llama(
    model_path="granite-4.2-3b-Heretic-Q5_K_M.gguf",
    n_ctx=8192,
    chat_format="chatml",
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "user", "content": "Explain chain-of-thought prompting."},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=1024,
)

print(response["choices"][0]["message"]["content"])

---

Thinking Mode

The base model supports built-in chain-of-thought reasoning through <think>...</think> tags. This behaviour carries over to the GGUF files when you use a compatible runtime that preserves special tokens.

With llama.cpp you can prompt the model to reason explicitly:

<|im_start|>user
How many prime numbers are there between 1 and 50?<|im_end|>
<|im_start|>assistant
<think>

The model will then complete its reasoning trace before emitting a final answer.

---

Model Details

---

About the Source Model

granite-4.2-3b-Heretic is a representation-edited version of IBM Granite 4.2 3B produced with the Heretic workflow. The purpose of the edit is research and experimentation with model behaviour, refusal patterns, and instruction-following trade-offs. The weights are modified; this is not an official IBM release.

For full details on the editing methodology, intended use, limitations, and evaluation guidance, refer to the source model card:

• tinyopsec/granite-4.2-3b-Heretic

For the original base model documentation:

• ibm-granite/granite-4.2-3b
• Granite 4.2 Technical Blog
• Granite 4.2 GitHub

---

Important Limitations

This model has not been comprehensively evaluated after representation editing and may differ from the base model in refusal behaviour, factual accuracy, coding performance, and output stability. Outputs should be reviewed by a human before use in any consequential application.

The model should not be relied upon as an autonomous authority or used as a substitute for professional medical, legal, or financial judgment.

---

License

Released under the Apache License 2.0, subject to the license and acceptable-use conditions of the base IBM Granite 4.2 3B model.