ausf-deberta-prompt-injection-v1

Fine-Tuned AUSF DeBERTa Prompt Injection Classifier

This model is a fine-tuned version of ProtectAI/deberta-v3-base-prompt-injection-v2 specifically optimized for the AUSF (App Security Gateway) to handle sophisticated prompt injections, adversarial overrides, and base64-obfuscated bypass attempts.

Key Architecture & Features

• Base Architecture: DeBERTa-v3-base (English-centric, relative position embeddings).
• Base64 Robustness: Injected with Base64 adversarial shards during fine-tuning, allowing the model to naturally identify encoded prompt injections with 99.5% accuracy without requiring a decoding pipeline.
• Low False Positives: Tuned with hard negatives (conversations discussing hacking conceptually, benign overrides) to prevent blocking legitimate developer questions.

---

Performance Evaluation & Benchmarks

The model was tested against Meta's Prompt-Guard-86M, ProtectAI's Base Model, and two GLiGuard-style models across both a standard unseen dataset and a hard adversarial synthetic validation set (1,700 samples).

1. HuggingFace Zero-Shot Evaluation (deepset/prompt-injections)

Public standard prompt injection test split (116 samples).

Note: Meta's Prompt-Guard-86M achieves 0.704 F1 only by flagging 80.4% of benign user queries as attacks, making it unusable in production.

---

2. Adversarial Synthetic Benchmarks

Custom adversarial evaluation set containing 1,700 samples of Base64 obfuscation (ENCODING_BYPASS) and confusing security context queries (BENIGN_SECURITY / BENIGN_OVERRIDE).

---

Key Conclusions

1. Natively Handles Obfuscation: The fine-tuned model detects raw Base64 prompt injections with 99.5% accuracy, completely removing the need for pre-decoding pipeline filters.
2. Superior Accuracy & Safety: Reduces overall false positive rates on complex/confusing user prompts to 18.8% (compared to ProtectAI's 46.3% and Meta's 82.9%).