granite-switch-3b-prev

Granite Switch 4.1 3B Preview

Model Summary:

Granite Switch 4.1 3B Preview is a modular LLM built on IBM Granite 4.1 3B with embedded adapters from the Granite Libraries collection. A single checkpoint supports multiple specialized capabilities — RAG, safety, explainability, and more — that are activated on demand via control tokens in the chat template.

For full details on model composition and adapter configuration, see BUILD.md.

Base Model: ibm-granite/granite-4.1-3b (3B params, 128K context) Adapters: 12 adapters from 3 Granite Libraries (granitelib-rag-r1.0, granitelib-core-r1.0, and granitelib-guardian-r1.0 License: Apache-2.0 Release Date: May 5th, 2026 Backends: HuggingFace Transformers, vLLM Automatically Composed with: granite-switch

Granite Switch is also available in granite-switch-8b-4.1 preview and granite-switch-30b-4.1-preview — each with its own model card.

Motivation:

Traditional multi-task LLM deployments require either separate model copies per capability (multiplying memory and compute) or weight merging that permanently blends adapters and destroys task specialization. Granite Switch takes a different approach: independently trained activated LoRA adapters are embedded in a single checkpoint and dynamically selected at inference time via control tokens. KV cache normalization ensures adapters share no internal KV cache state as each adapter sees prior tokens only through the base model's representation. That way, adapters can build on each other's outputs, but never through another adapter's cached activations. This allows adapters to be developed independently and composed without accuracy loss. This makes it possible to implement LLM capabilities very efficiently and very accurately.

---

Included Adapters:

These adapters are best used via Mellea.

Core Library (ibm-granite/granitelib-core-r1.0)

Adapters for explainability, calibration, and constraint verification.

RAG Library (ibm-granite/granitelib-rag-r1.0)

Adapters for retrieval-augmented generation pipelines.

Guardian Library (ibm-granite/granitelib-guardian-r1.0)

Adapters for safety, factuality, and policy compliance.

---

Installation:

git clone https://github.com/ibm-granite/granite-switch.git
cd granite-switch

# Pick what you need:
pip install -e ".[compose]"   # Compose models with adapters
pip install -e ".[hf]"        # HuggingFace inference
pip install -e ".[vllm]"      # vLLM inference
pip install -e ".[dev]"       # Everything

Requires Python 3.9+ and PyTorch 2.0+.

---

Model Contents:

---

Generation:

Using with Mellea

Mellea is the preferred way to run Granite Switch adapters in applications. It standardizes the interface for building with adapters like answerability checking, hallucination detection, requirement checker and harmful language detection easily and reliably. Constrained decoding and input/output pre-processing are handled automatically, improving accuracy and reliability. When running Granite Switch models through Mellea, embedded adapters function as high-level API calls. This allows you to use direct operations instead of raw prompt engineering.

pip install mellea

Answerability check

from mellea.backends.openai import OpenAIBackend
from mellea.formatters import TemplateFormatter
from mellea.stdlib.components import Document, Message
from mellea.stdlib.components.intrinsic import rag
from mellea.stdlib.context import ChatContext

SWITCH_MODEL_ID = "ibm-granite/granite-switch-4.1-3b-preview"

backend = OpenAIBackend(
    model_id=SWITCH_MODEL_ID,
    formatter=TemplateFormatter(model_id=SWITCH_MODEL_ID),
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
    load_embedded_adapters=True,
)

context = ChatContext().add(Message("assistant", "Hello there, how can I help you?"))
question = "What is the square root of 4?"
documents = [Document("The square root of 4 is 2.")]

result = rag.check_answerability(question, documents, context, backend)
print(f"Answerability: {result}")

Requirement check

from mellea.stdlib.components.intrinsic import core

context = ChatContext().add(
    Message("user", "Invite for an IBM office party.")
).add(
    Message("assistant", "Dear Team, you are cordially invited to a team social...")
)

result = core.requirement_check(context, backend, requirement="Use a professional tone.")
print(f"Requirements Satisfied: {result}")  # float between 0.0 and 1.0

Guardian core (safety detection)

from mellea.stdlib.components.intrinsic import guardian

context = ChatContext().add(
    Message("user", "How can I hack my friend's email?")
)

score = guardian.guardian_check(context, backend, criteria="harm", target_role="user")
verdict = "Risk detected" if score >= 0.5 else "Safe"
print(f"Score: {score:.4f}  ({verdict})")

See the mellea examples/ directory for more examples, including manual adapter loading.

The following examples demonstrate low-level adapter invocation via the HuggingFace and vLLM backends directly. Check Granite Switch For additional tutorials.

HuggingFace Inference

import granite_switch.hf  # Register the model architecture

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-switch-4.1-3b-preview", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-switch-4.1-3b-preview")

Activate an adapter via the chat template

messages = [
    {"role": "assistant", "content": "Hello there, how can I help you?"},
    {"role": "user", "content": "What is the square root of 4?"},
]
documents = [{"doc_id": "1", "text": "The square root of 4 is 2."}]

prompt = tokenizer.apply_chat_template(
    messages,
    documents=documents,
    adapter_name="answerability",   # activate the answerability adapter
    add_generation_prompt=True,
    tokenize=False,
)

outputs = model.generate(**tokenizer(prompt, return_tensors="pt").to(model.device))
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# => "answerable"

No adapter (base model behavior)

prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)

vLLM Inference

Start the OpenAI-compatible server:

pip install -e ".[vllm]"

python -m vllm.entrypoints.openai.api_server \
  --model ibm-granite/granite-switch-4.1-3b-preview \
  --port 8000

Call adapters via the API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="ibm-granite/granite-switch-4.1-3b-preview",
    messages=[
        {"role": "assistant", "content": "Hello there, how can I help you?"},
        {"role": "user", "content": "What is the square root of 4?"},
    ],
    extra_body={
        "documents": [{"doc_id": "1", "text": "The square root of 4 is 2."}],
        "chat_template_kwargs": {"adapter_name": "answerability"},
    },
    max_completion_tokens=6,
)
print(response.choices[0].message.content)
# => "answerable"

Or with curl:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ibm-granite/granite-switch-4.1-3b-preview",
    "messages": [
      {"role": "assistant", "content": "Hello there, how can I help you?"},
      {"role": "user", "content": "What is the square root of 4?"}
    ],
    "documents": [{"doc_id": "1", "text": "The square root of 4 is 2."}],
    "chat_template_kwargs": {"adapter_name": "answerability"},
    "max_completion_tokens": 6
  }'

---

How It Works:

Granite Switch uses coarse-grained expert switching — one adapter is active across all layers for a contiguous span of tokens. A lightweight switch layer (standard attention) detects control tokens in the input and produces per-position adapter indices that tell every decoder layer which LoRA weights to apply.

Input Tokens: ["Tell", "me", "about", "<|answerability|>", ...]
                              │
                    ┌─────────▼──────────┐
                    │    SWITCH LAYER    │
                    │  Detects control   │
                    │  tokens → indices  │
                    └─────────┬──────────┘
                              │
                 adapter_indices: [0, 0, 0, 4, 4, ...]
                              │
                    ┌─────────▼──────────┐
                    │  DECODER LAYERS    │
                    │  Base weights +    │
                    │  LoRA[index]       │
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │     LM HEAD        │
                    └────────────────────┘

Key properties:

• KV cache normalization — each adapter sees only the base model's KV cache, never another adapter's internal state
• No joint training — adapters are developed, tested, and published independently
• Single checkpoint — one file works with both HuggingFace and vLLM, no conversion needed
• Zero code changes — adapter selection happens entirely through the chat template

---

Model Architecture:

Base Model

Adapter Activation

Each adapter is activated by passing its name to the chat template using an argument. The template inserts the appropriate control token automatically — callers just pass adapter_name.

---

Ethical Considerations and Limitations:

This model inherits the safety profile of the base Granite 4.1 model. The Guardian Library adapters (guardian core, factuality detection/correction, policy guardrails) provide additional safety layers but are not a substitute for application-level safety testing. Deployers should:

• Test adapter behavior on their specific use cases before production deployment
• Apply appropriate content filtering for their domain
• Monitor adapter outputs, especially for safety-critical applications
• Use the uncertainty adapter to assess model confidence on important decisions

---

Requirements:

---

Citation:

@software{granite_switch,
  title  = {Granite Switch: Coarse-Grained Expert Switching for LLMs},
  author = {IBM Research},
  year   = {2026},
  url    = {TBD}
}

See also: Activated LoRA: Fine-tuned LLMs for Intrinsics (NeurIPS 2025).

---

License

Granite Switch has an Apache-2.0 license, as found in the LICENSE file.