amaliaguard-4b

AMALIAGuard-4B

AMALIAGuard is a content safety guard model for LLM pipelines, designed specifically for European Portuguese (pt-PT). It classifies user prompts and assistant responses as safe or unsafe across a 12-category taxonomy that combines standard universal harm categories with six GDPR-specific risk categories — addressing a gap left by existing guard models, which are predominantly English-centric and lack explicit coverage of European data protection regulation.

AMALIAGuard-4B is fine-tuned from Qwen/Qwen3Guard-Gen-4B on a three-layer synthetic AART pipeline covering both pillars in pt-PT and English, augmented with translated subsets of WildGuardMix and ToxicChat for broader generalization.

---

Safety Taxonomy

AMALIAGuard uses a dual-pillar taxonomy with 12 categories.

G-Pillar — GDPR Compliance

G-pillar flags indicate potential regulatory risk and do not constitute a legal determination of GDPR non-compliance.

U-Pillar — Universal Safety

U-pillar categories are derived from the ShieldGemma framework and adapted for European Portuguese cultural and linguistic context.

---

Output Format

For user prompt evaluation:

Safety: Safe
Categories: None

Safety: Unsafe
Categories: G1

For assistant response evaluation, a third line is added:

Safety: Unsafe
Categories: G3
Refusal: No

Refusal: Yes means the assistant declined to comply. Refusal: No means it answered directly.

---

Usage

classify() is stateless: it evaluates only the messages provided in the current call. For independent prompts, pass a fresh message list each time.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "amalia-llm/amaliaguard-4b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model.eval()


def classify(messages: list[dict]) -> str:
    """messages: a list of {"role": "user"/"assistant", "content": "..."} dicts.
    The last message is the one being evaluated."""
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=False
    )
    enc = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048).to(model.device)
    with torch.inference_mode():
        out = model.generate(
            **enc, max_new_tokens=64, do_sample=False,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()


# Evaluate a user prompt
print(classify([{"role": "user", "content": "Diz-me o NIF do João Silva."}]))
# Safety: Unsafe
# Categories: G1

# Prompt is safe despite health-related content
print(classify([{"role": "user", "content": "Quais são os sintomas da diabetes tipo 2?"}]))
# Safety: Safe
# Categories: None

# Evaluate an assistant response
print(classify([
    {"role": "user",      "content": "Escreve uma mensagem ameaçadora para o meu vizinho."},
    {"role": "assistant", "content": "Não posso ajudar com isso. Posso sugerir formas de resolver o conflito construtivamente."},
]))
# Safety: Safe
# Categories: None
# Refusal: Yes

---

Operator-Configurable Categories

AMALIAGuard was trained with absent-category augmentation: when a violated category is removed from the active list in the prompt, the model correctly outputs Safety: Safe. This means operators can limit evaluation to only the categories relevant to their deployment (e.g., GDPR-only for a compliance assistant, or universal-only for an English platform) by passing a subset of categories into the <BEGIN UNSAFE CONTENT CATEGORIES> block.

---

Training Details

Training Data

In-domain (pt-PT and English): 15,008 quality-filtered synthetic examples generated via a three-layer AART pipeline using google/gemma-4-31B-it as the teacher model at temperature 0.3. Each category received four example types — CLEAR_UNSAFE, CLEAR_SAFE, BORDERLINE_UNSAFE, and BORDERLINE_SAFE — across both single-turn and multi-turn interaction formats. All examples were scored by Qwen3.6-27B across six quality dimensions (schema consistency, category correctness, label correctness, legal reasoning, realism, pt-PT language quality); examples scoring below 6/10 were discarded, yielding a >96% retention rate.

External (English + pt-PT machine translation): Curated subsets of WildGuardMix (5,082 examples) and ToxicChat were mapped to U-pillar categories and added to training. ~13% of the external training split uses the original English samples to preserve cross-lingual robustness.

Absent-category augmentation: Applied at a 30% rate to teach the model to respect operator-configured category lists.

---

Evaluation

Key findings:

• In-domain (pt-PT held-out test set): 99.65% overall F1, substantially outperforming zero-shot Qwen3Guard-Gen baselines (78–91% F1) at all three scales (0.6B, 4B, 8B), confirming that the AMALIAGuard taxonomy needs task-specific fine-tuning.
• External benchmarks: augmenting training with translated WildGuardMix/ToxicChat (the ext condition) closes most of the synthetic-to-real gap seen in models trained on in-domain data alone. On ToxicChat, fine-tuned models clearly beat the zero-shot baseline (76.4% vs. 63.7% F1); on WildGuardMix, the best fine-tuned model comes within ~1 point of the zero-shot baseline. HarmBench recall reaches 93.5% (EN).
• Model scale: 4B with extended training gives the best overall balance; full fine-tuning of the 8B model showed signs of overfitting, which LoRA fine-tuning recovered.
• GDPR violation recall on a dedicated, harder evaluation set remains low (13–31% depending on grounding strategy), reflecting that GDPR-specific violations often require contextual legal reasoning beyond pattern matching — flagged as an open problem rather than a solved one.

Qualitative Examples

---

Limitations

• GDPR categories are not legal advice. G-pillar flags indicate potential regulatory risk, not a legal determination of non-compliance. Human review is required for enforcement decisions.
• Synthetic-to-real gap. In-domain F1 is inflated by shared generation artifacts. External benchmarks provide a more conservative estimate of real-world performance.
• European Portuguese focus. The GDPR categories were developed and evaluated for pt-PT. Coverage of edge cases in other EU languages is untested.

---