aegis-shield-one-finance-1.0-3B-Performance

aegis-shield-one-finance-1.0-3B — Performance

Performance excerpt for aegis-shield-one-finance-1.0-3B, a finance- and Korean-specialized safety classifier fine-tuned from mistralai/Shieldstral-1.0-3B.

A 3B guardrail that matches or beats models up to 6× its size on published English safety benchmarks — and then adds Korean financial-risk classification that none of them can do at all.

---

The headline

Financial-crime and injection detection move from unusable (F1 21–29) to production-grade (F1 78–93) on identical inputs — at the same 3B parameter count.

---

1. The backbone already leads its weight class

Published by Mistral on official splits — prompt classification, F1 %:

3B vs. 20B / 8B / 4B / 12B / 9B. The backbone tops the table on ToxicChat and HarmBench against models up to 6× larger. aegis-shield-one builds on exactly this.

These are Mistral's published figures for the backbone. aegis-shield-one is not re-scored on those official splits — WildGuardTest, HarmBench and Aegis v2 are gated datasets — so it is not given a column here rather than being credited with numbers measured elsewhere.

---

2. What the fine-tune adds — measured head-to-head

Base Shieldstral-3B and aegis-shield-one run through the identical harness, corpora, policy, threshold and metric (FinKoGuard-Bench). Every delta below is a direct measurement, not a cross-paper comparison.

Korean financial regulatory risk — a capability, not an increment

The backbone scores zero — it does not emit the R1–R12 code at all. This is the single largest gap in the model, and it is the axis a Korean financial deployment is actually bought for.

Detection quality across every specialized axis

Finance saturates. All finance sets land at 0.999–1.000 ROC-AUC — 12,500 and 13,374-row benchmarks, not toy sets.

Behavioral attack trajectories

Base ranks attack trajectories below benign ones. aegis-shield-one separates them perfectly (n=7,869; 47 positives — clean separation on a small positive set).

English public anchors hold

The Korean/finance gains are not bought by giving up English general safety — ranking on the public anchors is preserved or better.

---

3. Quantization holds the numbers

GGUF builds re-scored end-to-end by the same harness — not assumed:

Q8_0 and Q5_K_M are effectively lossless — mean |ΔAUC| ≤ 0.0003. Full accuracy in 2.5 GB, CPU-servable via llama.cpp.

---

4. Built on proprietary data, not scraped benchmarks

72,339 training rows — 56.6 % built in-house. The gains above are not a public dataset re-mix; the axes that move are the axes this project built from scratch.

100 % of the finance and regulatory data — 27,703 rows — is self-built, because no public dataset covers Korean financial AI-risk typing:

The FinRisk R1–R12 corpus is the reason for the headline result. It is a 12,000-row in-house corpus split 70/30 at the cluster level — 6,837 rows trained, 5,027 rows across 39 held-out clusters kept as the benchmark. Cluster-level splitting means even paraphrase neighbours cannot leak across the boundary. The base model scores 0.00 on this axis; there is nothing public to train it on.

Clean-evaluation guarantee

Exact-match overlap between training data and every evaluation set is 0 — normalized-hash, turn-level for multi-turn, checked against all 28 corpora before training. Public anchors (BeaverTails, ToxicChat) are used train-split only, so the ToxicChat and BeaverTails numbers in §2 are measured on unseen rows.

Scope of this page

This is a performance excerpt. It reports measured results only — no figure here is estimated or carried over from a different benchmark. The complete model card carries the inference contract, the policy prompts, and the trade-offs this specialization costs — notably a Korean multi-turn regression that makes base Shieldstral the right choice for multi-turn traffic. Read it before deploying:

• Full model — yatavent/aegis-shield-one-finance-1.0-3B
• GGUF builds — yatavent/aegis-shield-one-finance-1.0-3B-GGUF
• Harness — FinKoGuard-Bench

Metrics: ROC-AUC and F1 at the fixed 0.5 threshold, general policy unless a row names another. Peer-model figures in §1 are Mistral's published numbers for the backbone, reproduced for positioning.