gemma-3-4b-scambreaker-adapter

Scambreaker-Gemma-3-4B

Fine-tuned Gemma-3-4b-it on a bilingual (English/Russian) scam-risk analysis instruction dataset for local-first scam defense for non-technical people, built for the AutoScientist Challenge — Personal-Finance category.

Model Details

Why Personal Finance

Scams are a direct, first-party financial-loss problem: wire fraud, gift-card and crypto payments, and stolen banking credentials move money out of a victim's account in minutes, and are rarely recoverable afterward. The highest-risk targets — older adults, non-native speakers, and anyone unfamiliar with a scam's exact playbook — are also the least equipped to catch the manipulation in the moment. This model reads one suspicious message and, before any money or credentials move, tells the person in plain language whether it's dangerous, what the attacker is trying to extract, and the single safest next step. That is a personal-finance protection tool in the most literal sense: it exists to stop money from leaving a real person's account into a scammer's.

Model Demo

Click to image below or press here - Demo Video. Try the Model Demo to see the model analyze potential scam messages and explain how to avoid them.

Model Description

Scambreaker-Gemma-3-4B is a LoRA fine-tuned version of Gemma-3-4b-it, adapted to analyze a single suspicious message (SMS, email, DM) and return a validated JSON safety card: a risk level, the scam type, the manipulation tactics used, a compact "scam DNA" breakdown, the single safest next action, a message to forward to a trusted person, and a short plain-language summary.

The base Gemma-3-4B model was chosen for its multilinguality and suitability for a small fine-tuning dataset. No additional data was used during fine-tuning; the goal was solely to teach the model the JSON output format and the scam-analysis task itself, not to expand its knowledge.

• Developed by: Sultanov Danial
• Model type: Causal LM, LoRA fine-tuned (text-decoder only)
• Dataset: dokster/adaption-scam-dataset-ru-en-v1
• Language(s): English, Russian
• License: Gemma Terms of Use
• Finetuned from: google/gemma-3-4b-it
• Training platform: Adaption Labs

Uses

Direct Use

Given one suspicious message (English or Russian), it returns JSON scam-analysis:

• risk_level: dangerous / suspicious / needs_check / safe
• scam_type: fixed-vocabulary label, e.g. credential_theft, job_scam, tech_support
• tactics: up to 3 short manipulation-tactic strings
• scam_dna: impersonates / pressure / ask / risk — short descriptive phrases, in the message's language
• safest_action, trusted_person_message, summary: plain-language guidance, in the message's language

Intended as decision support for a non-technical person deciding whether to click, reply, or call — not an autonomous blocker.

Out-of-Scope Use

• Not legal, financial, or cybersecurity advice
• Not a spam/scam filter for production email or SMS pipelines without additional evaluation on real-world traffic (training data is synthetic — see Bias, Risks, and Limitations)
• Not evaluated for languages other than English and Russian
• Should never be wired to auto-block, auto-delete, or auto-reply without a human in the loop

Bias, Risks, and Limitations

• Synthetic training data. Messages are template-generated, not collected real-world scams; phrasing and scenario mix may not match real traffic.
• US-centric scenarios, including in the Russian-language examples (USPS, IRS, DMV, Zelle) — the dataset does not reflect scam patterns specific to Russian-speaking regions.

Recommendations

Always present output as guidance to slow down and verify through an official channel — never as a definitive verdict, and never surface an action that tells the user to click the suspicious link or call the number in the message itself.

How to Get Started with the Model

from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForImageTextToText.from_pretrained(
    "google/gemma-3-4b-it",
    dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "dokster/gemma-3-4b-scambreaker-adapter")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")

SYSTEM_PROMPT = "..."  # full JSON schema + safety-rules prompt (see repo)
message = "USPS: Your package is held. Verify account details now: https://... Ref 1000."

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": SYSTEM_PROMPT},
     {"role": "user", "content": f"/no_think\n\nAnalyze this message for scam risk:\n\n{message}\n\nReturn only the JSON object."}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Note: google/gemma-3-4b-it is a gated model. Request access before loading

Training Details

Training Data

2,324 examples (1,200 English + 1,124 Russian), derived from the synthetic scam-message generator. English examples are the deterministic generator output. Russian examples were produced by translating only the natural-language fields (never the fixed risk_level/scam_type/tactics labels) with an LLM under strict brand-name and format-preservation rules, then mapped back across the full row set. See dokster/adaption-scam-dataset-ru-en-v1 for full dataset documentation.

Training Procedure

Training Hyperparameters

• Training regime: bf16 mixed precision
• Algorithm: LoRA (Supervised Fine-Tuning)
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.0
• Target modules: all-linear
• Learning rate: 1e-5
• LR scheduler: cosine, warmup ratio 0.05
• Epochs: 2
• Weight decay: 0.0
• Gradient clipping: 1.0
• Held-out validation: 5% random split, best checkpoint selected by eval loss

Evaluation

Testing Data, Factors & Metrics

Testing Data

This model achieves 94% winrate over base model. To approve this number I created special evaluation dataset. Evaluated on a "hard" synthetic eval suite — 632 English cases spanning boundary/calibration scenarios (credential theft, family impersonation, tech support, job scams, marketplace fraud, "wrong number" investment scams, and matched safe/benign controls), plus a 316 EN + 316 RU bilingual subset for a per-language read. This is a held-out synthetic suite, not real-world traffic — see Bias, Risks, and Limitations.

Metrics

• risk_level accuracy: exact match against the gold risk label
• dangerous → safe / needs_check: count of dangerous cases under-called — the safety-critical failure mode
• safe → over-flagged: count of benign messages incorrectly flagged, a usability failure mode
• scam_type / tactic recall: secondary structured-output quality metrics

Results

Full English hard suite (632 cases):

Bilingual EN/RU (316 + 316 cases):

Summary

Zero safety-critical misses on the full, reliable English suite — no dangerous case was ever mislabeled safe or needs_check, and no benign message was ever over-flagged.

Baseline Comparison: Fine-tuned vs. Base Model

To isolate what fine-tuning actually contributes, the base google/gemma-3-4b-it model (zero-shot, no LoRA adapter, same system prompt/schema) was evaluated on the identical full English hard suite (632 cases):

The base model already produces valid JSON and never misses a dangerous case outright — the system prompt alone gets it that far. What fine-tuning adds is accuracy and calibration: +30 points on risk classification, a ~34x jump in scam_type accuracy, and elimination of false alarms on benign messages (13 → 0). This is the clearest evidence that the LoRA training taught the model the task itself, not just the output format.

Citation

@misc{scam_detection_ru_en_2026,
  title        = {Scam-Risk Analysis: A Bilingual Structured-Output Dataset for Scam Defense},
  author       = {Sultanov Danial},
  year         = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/dokster/gemma-3-4b-scambreaker-adapter}},
  note         = {AutoScientist Challenge 2026 — Personal-Finance Category}
}

Model Card Authors

Danial Sultanov - Nazarbayev University, BSc Mathematics