teliti-job-scam-mdistilbert

TELITI — Job Advertisement Integrity Scoring

Fine-tuned distilbert-base-multilingual-cased that scores how likely an Indonesian job advertisement is to be a scam, so a jobseeker can check it before sending a CV and personal data.

Part of TELITI (Teknologi Evaluasi Lowongan dan Integritas). Full system — API, rule engine, explanations, web app: https://github.com/Gigvo/TELITI

What is in this repository

Results, stated honestly

Two evaluations on two datasets. The numbers are not comparable to each other — PR-AUC scales with how common the positive class is, so the higher number is not evidence of a better model.

EMSCAD test split — the honest number

Scored once, on a split untouched by any earlier decision.

Quote 0.7883, not the validation figure. The validation score was 0.8669, but the checkpoint was selected on validation, which makes that split an optimistic estimate. The transformer drops 0.0786 PR-AUC from validation to test; the baseline, which involved no such selection, drops only 0.0118.

The transformer does not beat a TF-IDF baseline on English EMSCAD. It is used because TF-IDF trained on English cannot process Indonesian at all, and Indonesian is the deployment domain — not because of this number.

EMSCAD validation (English, 1,788 items, 4.8% scams)

Reported for completeness. The checkpoint was selected here, so these are optimistic — see the test figures above.

Accuracy is 98.3% and is misleading — at 4.8% prevalence, predicting "not a scam" for everything scores 95.2%.

Indonesian holdout (195 items, 36.4% scams)

The rule layer made things worse and is disabled in the product. Concept paper §3.6 names false positives against real businesses as the expensive error, and the rules multiplied them by five.

Usage

import json
import math

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "Gigvo/teliti-job-scam-mdistilbert"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

# Raw logits are NOT probabilities here. Apply the deployment calibrator.
with open(hf_hub_download(REPO, "calibrator_deployment.json")) as fh:
    calibrator = json.load(fh)

text = (
    "Dibutuhkan admin online, gaji Rp9.000.000, tanpa pengalaman. "
    "Wajib transfer biaya administrasi Rp250.000."
)

with torch.no_grad():
    encoded = tokenizer(text, truncation=True, max_length=256, return_tensors="pt")
    logits = model(**encoded).logits

# The calibrator is fitted on the LOGIT MARGIN, not on softmax output.
margin = (logits[0, 1] - logits[0, 0]).item()
p_scam = 1 / (1 + math.exp(-(calibrator["a"] * margin + calibrator["b"])))

print(f"p(scam) = {p_scam:.3f}")
print(f"integrity score = {round((1 - p_scam) * 100)} / 100")

Verified output — scam example above scores 0, and a legitimate posting scores 91.

Note max_length=256. The model was fine-tuned at 512, but the product serves at 256 to keep occlusion-based explanation within its latency budget; use 256 to reproduce the product's numbers exactly.

Calibration is not optional

Raw softmax(logits) is not a usable probability here. The model was trained where scams are 4.8% of postings and deployed where they are ~36%. Without recalibration every Indonesian ad scored 93–100 out of 100 — technically ranked correctly, useless as a number a person can act on.

calibrator_deployment.json corrects that shift. Use it.

Training

Trained on EMSCAD (Employment Scam Aegean Dataset), then recalibrated on a manually annotated Indonesian holdout.

Intended use

For: helping a jobseeker decide whether to look more carefully at an advertisement before applying.

Not for: automated rejection of postings, moderation without human review, or any decision about a company or a person taken on the score alone.

Limitations

• Trained on English data. EMSCAD is English; Indonesian performance is estimated from a 195-item holdout. Indicative, not established.
• Did not beat a TF-IDF baseline on EMSCAD validation.
• EMSCAD is redacted. Emails, URLs and phone numbers were replaced with placeholders before publication, so the model never learned from real contact details — the exact signal that matters most in a real scam advertisement.
• Not a verdict. The score is a risk indicator. A high score is not proof of fraud and a low score is not a guarantee of safety.
• Ghost jobs are out of scope. Detecting advertisements for roles that do not exist is a separate problem and is not implemented.

Ethics

Concept paper §3.6 treats false positives against legitimate businesses as the expensive error, which is why the rule layer was disabled after measurement rather than kept because it was built. The product ships a visible disclaimer and an appeal route; appeals are reviewed by a person and are not used to retrain.

Citation

@misc{teliti2026,
  title  = {TELITI: Teknologi Evaluasi Lowongan dan Integritas},
  author = {Tim SateManaAjaDah},
  year   = {2026},
  note   = {Datathon 2026, Ristek Fasilkom UI},
  url    = {https://huggingface.co/Gigvo/teliti-job-scam-mdistilbert}
}