scope-guard-v2-4B-q-2609

ScopeGuard-2: A Governance SLM for Scope Classification

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/5f0b462819cb630495b814d7/3JLRdRmw2CDJM814FN-d4.png" alt="scopeguard-2-banner" width="1000"> </p> ScopeGuard-2 is the second generation of our small language model (SLM) for scope classification in AI governance. Like its predecessor, it is trained to make consistent decisions based on service policies, with the low latency needed to check every incoming request.

We retain the previous generation's focus on a small, specialized model. This is a deliberate design choice: keeping inference costs and latency low makes ScopeGuard practical to deploy as part of a production application's request handling. ScopeGuard-2 builds on that approach with improved classification accuracy and faster inference.

Table of contents

• What ScopeGuard-2 does
• Show me the code
• Model family
• Evaluation
• Inference speed
• Intended use
• Limitations
• Cite
• Contact

---

Resources

• Use ScopeGuard-2 with Orbitals: our Python library on GitHub

<!-- * **Learn more about ScopeGuard-2**: [link to the blog article] -->

• How we evaluate: unsafe-eval, our open-source framework for reproducible safety evaluation

What ScopeGuard-2 does

Given a user request, ScopeGuard-2 decides whether it falls within the AI service's scope or should be declined, redirected, or passed to a person. You define that scope through the service's supported use cases, allowed topics, and interaction rules.

ScopeGuard-2 can be used as an early gate in enterprise AI deployments to ensure requests are handled only when they match the intended purpose of the system. The new release, also identifies requests that call for a predefined reply or human oversight.

Given a conversation, ScopeGuard-2 assigns the last user message to one of seven classes, each mapped to a different action:

You can also request a one-sentence rationale, passages from your description that support the decision, and a suggested reply in the user's language.

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/5f0b462819cb630495b814d7/pHN8cahWIxlpoZJNvqcoc.png" alt="scopeguard-2-overview-transparent" width="1200"> </p>

---

Show me the code

Install

pip install "orbitals[scope-guard-vllm]>=0.6.1"

# Or, to use transformers as a backend
# pip install "orbitals[scope-guard-hf]>=0.6.1"

Use

from orbitals.scope_guard_v2 import ScopeGuardV2

if __name__ == "__main__":
  sg = ScopeGuardV2(
      backend="vllm",
      model="principled-intelligence/scope-guard-v2-4B-q-2609",
  )
  
  ai_service_description = """
  You are a virtual assistant for a parcel delivery service.
  You can only answer questions about package tracking.
  Never respond to requests for refunds.
  """
  
  user_query = "If the package hasn't arrived by tomorrow, can I get my money back?"
  result = sg.validate(
      user_query,
      ai_service_description=ai_service_description,
      output_fields=["reasoning", "scope_class"],
  )
  
  print(f"Scope: {result.scope_class.value}")
  print(f"Reasoning: {result.reasoning}")
  
  # Scope: Restricted
  # Reasoning: The user is asking about getting a refund, which is explicitly forbidden by the service constraints.

output_fields accepts any subset of evidences, reasoning, scope_class, and suggested_response. The scope_class field is always included. When generating output, requesting scope_class alone gives the lowest latency and highest accuracy in our evaluation (see Inference speed and deployment).

Structured AI service description (recommended)

The structured description has dedicated fields for constraints, escalation criteria and predefined responses, which map directly onto Restricted, Human Oversight and Predefined Answer. ScopeGuard reads them more reliably than the same rules buried in free text.

from orbitals.scope_guard_v2 import ScopeGuardV2
from orbitals.types import AIServiceDescriptionV2, PredefinedResponse

if __name__ == "__main__":
  sg = ScopeGuardV2(
      backend="vllm",
      model="principled-intelligence/scope-guard-v2-4B-q-2609",
      output_fields=["scope_class"],
  )
  
  ai_service_description = AIServiceDescriptionV2(
      identity_role="PackAssist, a virtual assistant that helps customers track their parcels.",
      context="A parcel-delivery company. Users are customers awaiting or sending parcels.",
      functionalities=[
          "Retrieve tracking updates",
          "Explain tracking events and estimated delivery windows",
      ],
      constraints=["Never process or promise refunds"],
      escalation_criteria=["The customer reports a lost or stolen parcel"],
      predefined_responses=[
          PredefinedResponse(
              trigger="Asks for the customer service phone number",
              response="You can reach us at +1 555 0100, Mon-Fri 9-18.",
          ),
      ],
  )
  
  conversation = [
      {"role": "user", "content": "I ordered a package, tracking number 1234567890"},
      {"role": "assistant", "content": "It is in transit. What would you like to know?"},
      {"role": "user", "content": "If it doesn't arrive tomorrow, can I get a refund?"},
  ]
  
  result = sg.validate(conversation, ai_service_description=ai_service_description)
  print(f"Scope: {result.scope_class.value}")
  
  # Scope: Restricted

For conversations with multiple turns, ScopeGuard classifies the last user message using earlier turns as context. batch_validate classifies many conversations in one call, and AsyncScopeGuardV2 offers the same methods for async code.

If vLLM is unavailable on your machine, use the Transformers backend. It is slower, so prefer vLLM for serving:

sg = ScopeGuardV2(
    backend="hf",
    model="principled-intelligence/scope-guard-v2-4B-q-2609",
)

Probabilities instead of labels

classify skips generation and returns a probability for each class from a single forward pass. It is the fastest way to run ScopeGuard and lets you route requests using a confidence threshold:

result = sg.classify(conversation, ai_service_description=ai_service_description)
print(result.scope_class.value, result.confidence)
print(result.probabilities)

Serve it

orbitals can start vLLM and an HTTP API in front of it:

pip install "orbitals[scope-guard-serve]>=0.6.1"
orbitals scope-guard-v2 serve principled-intelligence/scope-guard-v2-4B-q-2609 --port 8000

Then call it from Python with ScopeGuardV2(backend="api", api_url="http://localhost:8000"), or directly over HTTP. See the Orbitals documentation for the available endpoints and configuration options.

---

ScopeGuard model family

• scope-guard-v2-4B-q-2609 (this model) Open-weight ScopeGuard-2, based on Qwen3.5-4B.

• ScopeGuard-2 Pro Our most capable closed-source model, available through our hosted API.

• scope-guard-4B-q-2601 and scope-guard-4B-g-2601 The previous generation, based on Qwen3-4B-Instruct-2507 and gemma-3-4b-it, released under Apache-2.0.

What's new in ScopeGuard-2

• Seven classes instead of five. Predefined Answer and Human Oversight identify requests that your application should handle with a fixed reply or pass to a person.
• Conversation context. The last user message is classified with the rest of the conversation as context.
• A richer service description, with constraints, escalation criteria and predefined responses as separate fields.
• Choose what the model returns: the class alone or with a rationale, supporting evidence, and a suggested reply. You can also request a probability for each class.
• Faster and more accurate. Against scope-guard-4B-q-2601, 1.3 percentage points higher mean balanced accuracy on the six benchmarks containing both safe and unsafe prompts, with about one-seventh the median latency (34 vs 241 ms).

---

How we evaluated ScopeGuard

We compared both ScopeGuard-2 models against:

• General-purpose models: GPT-6.1 Sol, Claude Opus 5.5, Gemini 4 Argon, and Jev 1.13.0
• Safety classifiers: NVIDIA Nemotron-3.5-Content-Safety and Shieldstral-1.0-3B
• Our previous generation: scope-guard-4B-q-2601, scope-guard-4B-g-2601, and ScopeGuard Pro

We used unsafe-eval, our open-source framework for reproducible AI safety evaluation. Each model labels each prompt as safe or unsafe. For ScopeGuard, Restricted and Human Oversight count as unsafe. Every other class, including Out of Scope, counts as safe.

We report balanced accuracy, the mean of the percentage of unsafe prompts caught and the percentage of safe prompts allowed through. This gives safe and unsafe prompts equal weight regardless of their proportions in each benchmark.

Tasks

1. General safety classification Detect whether a user request violates generic safety policies (toxicity, violence, illegal activity, abuse).

2. Custom safety classification Detect whether a user request violates policies defined by the service owner, which differ across products and organizations.

---

Results

Balanced accuracy (%) on six public benchmarks. Aegis 2.0, OpenAI Moderation, ToxicChat, and WildGuardMix cover general safety. CoSApien and DynaGuardrail test whether the model follows explicit policies supplied as part of the input.

ScopeGuard-2 4B scores 87.2% on average, compared with 85.9% for the previous Qwen-based release. Its score is within half a point of Claude Opus 5.5 at 87.6% and above GPT-6.1 Sol at 86.8%, while its median response time is 34 ms compared with 2,976 ms and 2,786 ms. ScopeGuard-2 Pro and Jev have the highest average scores in this comparison, at 88.3% and 88.0%, respectively.

On custom policies, ScopeGuard-2 4B scores 93.7% on CoSApien and 90.8% on DynaGuardrail. ScopeGuard-2 Pro scores 94.6% and 90.6%, respectively. These results are comparable to the best general-purpose models in this evaluation.

On SimpleSafetyTests and HarmBench, which contain only unsafe prompts and are excluded from the six-benchmark mean above, ScopeGuard-2 4B catches 99.0% and 98.4% of prompts, respectively.

---

Inference speed and deployment

A guardrail adds latency to every user interaction. We measured ScopeGuard-2 4B locally on a single NVIDIA GeForce RTX 5090 (32 GB), processing one request at a time. Hosted API response times include the network round trip and depend on network conditions and serving load.

The table reports median response times across all timed requests on the six benchmarks above, including CoSApien and DynaGuardrail. ScopeGuard-2 results use classification mode.

<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/principled-intelligence/scope-guard-v2-4B-q-2609/resolve/main/assets/accuracy-vs-latency-dark.png"> <img src="https://huggingface.co/principled-intelligence/scope-guard-v2-4B-q-2609/resolve/main/assets/accuracy-vs-latency-light.png" alt="Mean balanced accuracy versus median response time for ScopeGuard-2 and the other models in the comparison" width="800"> </picture> </p>

Output configuration

Generating additional fields takes longer. For ScopeGuard-2 4B, generating the class alone takes 87 ms, compared with 34 ms in classification mode. Adding reasoning raises the median to 338 ms.

These measurements use the same six benchmarks. Classification mode and generating the class alone have nearly identical accuracy. Requesting additional fields reduced accuracy in this evaluation, though explanations and supporting evidence can help you review decisions.

---

Intended use

ScopeGuard is intended for:

• Scope enforcement for customer-facing assistants: answer, decline, send a fixed reply, or hand off to a person, depending on the class
• Inline guardrailing for agentic systems (pre-checks before tool execution)
• Enterprise governance layers where behavior must follow explicit boundaries and policies
• Analytics and routing pipelines, where the class and its rationale support monitoring and reporting

---

Limitations

• ScopeGuard-2 is designed for governance decisions, not for open-ended generation.
• Training data is predominantly English. suggested_response follows the user's language, but classification quality in other languages has not been measured for ScopeGuard-2.
• Performance on safety benchmarks can be sensitive to prompting and evaluation conditions; strong results on one benchmark may not transfer automatically to all setups.

<details> <summary>Using ScopeGuard-2 without <code>orbitals</code></summary>

orbitals builds the prompt, restricts decoding to the requested JSON keys and parses the result. Without the library you do all three yourself. The prompt has to match the training prompt byte for byte, because a different one lowers accuracy and raises no error.

The code below renders the prompt with the model's tokenizer and sends it to the completions endpoint of a vLLM server. This is the same request the orbitals API backend makes.

pip install vllm openai
vllm serve principled-intelligence/scope-guard-v2-4B-q-2609 --port 8000

import json

from huggingface_hub import hf_hub_download
from openai import OpenAI
from transformers import AutoTokenizer

MODEL = "principled-intelligence/scope-guard-v2-4B-q-2609"
SYSTEM_PROMPT = open(hf_hub_download(MODEL, "system_prompt.txt"), encoding="utf-8").read()
tokenizer = AutoTokenizer.from_pretrained(MODEL)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

CLASSES = [
    "Directly Supported",
    "Potentially Supported",
    "Predefined Answer",
    "Human Oversight",
    "Out of Scope",
    "Restricted",
    "Chit Chat",
]

# The model emits the keys in this order, whatever order you request them in.
FIELD_SCHEMAS = {
    "evidences": {"anyOf": [{"type": "array", "items": {"type": "string"}}, {"type": "null"}]},
    "reasoning": {"type": "string"},
    "scope_class": {"enum": CLASSES},
    "suggested_response": {"anyOf": [{"type": "string"}, {"type": "null"}]},
}


def render_prompt(description: str, conversation: list[dict], fields: list[str]) -> str:
    *history, last = conversation
    dump = "".join(f"{m['role'].upper()}:\n{m['content']}\n\n" for m in history)
    dump += f"LAST MESSAGE (USER):\n{last['content']}\n"
    user_turn = (
        f"**START OF THE AI SERVICE DESCRIPTION**\n\n{description}\n\n"
        "**END OF THE AI SERVICE DESCRIPTION**\n\n\n"
        f"**START OF THE CONVERSATION DUMP**\n\n{dump}\n\n"
        "**END OF THE CONVERSATION DUMP**\n\n\n"
        f"**REQUESTED OUTPUT FIELDS**\n\n{json.dumps(fields)}"
    )
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_turn},
    ]
    return tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
    )


def validate(description: str, conversation: list[dict], output_fields=("scope_class",)) -> dict:
    fields = [f for f in FIELD_SCHEMAS if f in output_fields or f == "scope_class"]
    schema = {
        "type": "object",
        "properties": {f: FIELD_SCHEMAS[f] for f in fields},
        "required": fields,
        "additionalProperties": False,
    }
    completion = client.completions.create(
        model=MODEL,
        prompt=render_prompt(description, conversation, fields),
        temperature=0,
        max_tokens=1024,
        extra_body={"structured_outputs": {"json": schema}},
    )
    return json.loads(completion.choices[0].text)

Here it is with the structured description from Show me the code:

description = json.dumps(
    {
        "identity_role": "PackAssist, a virtual assistant that helps customers track their parcels.",
        "context": "A parcel-delivery company. Users are customers awaiting or sending parcels.",
        "knowledge_scope": None,
        "functionalities": [
            "Retrieve tracking updates",
            "Explain tracking events and estimated delivery windows",
        ],
        "constraints": ["Never process or promise refunds"],
        "predefined_responses": [
            {
                "trigger": "Asks for the customer service phone number",
                "response": "You can reach us at +1 555 0100, Mon-Fri 9-18.",
            }
        ],
        "escalation_criteria": ["The customer reports a lost or stolen parcel"],
        "response_guidelines": None,
    },
    separators=(",", ":"),
    ensure_ascii=False,
)

conversation = [
    {"role": "user", "content": "I ordered a package, tracking number 1234567890"},
    {"role": "assistant", "content": "It is in transit. What would you like to know?"},
    {"role": "user", "content": "If it doesn't arrive tomorrow, can I get a refund?"},
]

print(validate(description, conversation, ["reasoning"]))

# {'reasoning': 'The user is asking about the possibility of receiving a refund, which directly
#  violates the constraint that the AI must never process or promise refunds.', 'scope_class': 'Restricted'}

Common pitfalls

• A structured description must match AIServiceDescriptionV2.model_dump_json() exactly: all eight keys in order, null for unset ones, dumped as in the example.
• The last message in the conversation must come from the user.
• Set max_tokens. The completions endpoint defaults to 16 tokens and cuts the JSON short.
• With transformers, pass eos_token_id=tokenizer.eos_token_id to generate, or the model keeps writing after the JSON.

Class probabilities

classify requests scope_class alone, appends {"scope_class": " to the prompt and reads the log-probabilities of the next token. The seven class names each start with a different token, so normalizing over those tokens gives one probability per class:

import math

PREFIX = '{"scope_class": "'
prefix_len = len(tokenizer.encode(PREFIX, add_special_tokens=False))
first_tokens = {
    c: tokenizer.decode(tokenizer.encode(PREFIX + c, add_special_tokens=False)[prefix_len])
    for c in CLASSES
}


def classify(description: str, conversation: list[dict]) -> dict[str, float]:
    completion = client.completions.create(
        model=MODEL,
        prompt=render_prompt(description, conversation, ["scope_class"]) + PREFIX,
        temperature=0,
        max_tokens=1,
        logprobs=20,
    )
    top = completion.choices[0].logprobs.top_logprobs[0]
    scores = {c: math.exp(top.get(t, -math.inf)) for c, t in first_tokens.items()}
    total = sum(scores.values())
    return {c: s / total for c, s in scores.items()}


print({c: round(p, 4) for c, p in classify(description, conversation).items()})

# {'Directly Supported': 0.0066, 'Potentially Supported': 0.0046, 'Predefined Answer': 0.004,
#  'Human Oversight': 0.0003, 'Out of Scope': 0.0005, 'Restricted': 0.9839, 'Chit Chat': 0.0001}

</details>

---

Cite ScopeGuard

If you use ScopeGuard in academic work or evaluations, please cite it. Our technical report is coming soon.

---

Integration and contact

ScopeGuard-2 is released under the terms in LICENSE. If you are thinking about integrating ScopeGuard into your pipeline, or want to learn more about ScopeGuard-2 Pro, write to [email protected].

---

Principled Intelligence