saarthi-default-prediction

SAARTHI — MSME Default Prediction

Credit default-prediction models trained entirely on public datasets for IDBI Innovate 2026 (Track 04, Default Prediction Model). No bank data was used at any point.

The headline design choice: SAARTHI ships pre-trained and then fine-tunes on a lender's own book, rather than cold-starting on every upload.

Methodology — why these numbers are trustworthy

Every metric below comes from a strict three-way split:

This matters: a common shortcut fits the probability calibrator and then measures calibration error on that same fold, which drives ECE artificially toward zero. Here the calibrator never sees the test fold, so the reported ECE is a real out-of-sample calibration estimate.

Per-dataset specialist models

Full native feature set per corpus. Ensemble of LightGBM + XGBoost + CatBoost, isotonic-calibrated.

• sba — US Small Business Administration loans - closest public analogue to MSME lending
• lending_club — 2007-2018 consumer loans, charge-off label
• home_credit — Home Credit 2018 application + bureau aggregates
• gmsc — Give Me Some Credit, 90+ DPD within 2 years
• taiwan — Taiwan credit-card default, 6-month repayment panel
• german — UCI German Credit, 1000 rows
• berka — PKDD'99 Czech bank, pre-origination transaction aggregates
• amex — American Express monthly statement panel
• hc2024 — Home Credit 2024 model-stability competition

Pooled global model (ships with the app)

Trained across corpora in a shared 15-field canonical credit vocabulary so it can score any loan book that maps onto it.

• Test AUC: 0.8605
• PR-AUC: 0.5795
• ECE: 0.0018 · Brier: 0.0868
• Test rows: 270,421
• Corpora pooled: sba, lending_club, home_credit, gmsc, taiwan, german, berka

Leave-one-dataset-out transfer — a negative result

Train on every corpus except one, then score the held-out corpus cold. This is the honest proxy for "will it transfer to a book it has never seen?" — the question that actually matters for deployment.

It does not transfer. Most hold-outs land at or below chance, which means the pooled model is not merely uninformative on an unseen corpus, it is anti-predictive: the feature→outcome relationships invert across lending domains.

Normalisation did NOT rescue it

The obvious hypothesis is scale mismatch — these corpora are denominated in USD, DM, NT$ and CZK, so a "loan amount" of 50,000 means different things in each. Converting every numeric feature to its within-corpus percentile rank tests that hypothesis directly, and it fails: mean transfer AUC moved 0.5486 → 0.5377, i.e. no better. SBA in particular degrades from 0.4103 to 0.2574.

The mechanism is therefore relational inversion, not scale. SBA is small-business lending, where a longer term and a larger SBA-guaranteed principal typically indicate a better-vetted, collateral-backed loan — the opposite of the consumer-credit corpora that dominate the pooled training set. Rank-normalising preserves that inverted ordering perfectly, which is why it cannot help.

Practical consequence: a single pooled "foundation" model for credit risk is not supported by this evidence. Domain-matched training plus fine-tuning on the lender's own book is the defensible architecture, which is what SAARTHI does.

Ablation — does transaction-as-language earn its place?

A CoLES-style contrastive encoder (GRU + InfoNCE over disjoint sub-sequence views) was pre-trained on unlabelled transaction streams, then its 256-d embedding was tested against the tabular features.

When the sequence encoder earns its place: on Berka (16 tabular features, raw bank transactions) it adds +0.027 AUC. On Amex (941 hand-engineered aggregates over the same statement data) it adds nothing — the aggregates already capture what the encoder learns. The embedding substitutes for feature engineering rather than adding to it; it is most valuable exactly where a lender has raw transaction streams but few curated fields, which is the realistic MSME case.

Note also that on Amex the 256-d embedding alone reaches 0.9156 AUC with no hand-engineered features at all, versus 0.9590 for 941 engineered columns.

Linear probe on frozen embeddings

Logistic regression on the frozen encoder output — measures how much default signal the unsupervised embedding captured on its own.

• amex: AUC 0.8717 (n=60,000)
• berka: AUC 0.7783 (n=682)

Known limitations

• Berka aggregates only pre-origination transactions. Using the full history yields AUC 1.0000 because 71% of an account's transactions occur after the loan date and encode the repayment behaviour that defines the label. The leaked figure is not reported here.
• Amex and Home Credit are Kaggle-competition datasets; they are used for research validation. The shipped global model is trained on the permissively-licensed core corpora.
• The pooled model uses a deliberately small shared vocabulary, so its AUC is lower than the specialists by construction. Its purpose is transfer, not peak in-corpus accuracy.
• No Indian MSME data was available; GST-behavioural fields are simulated against the schema IDBI published, not learned from real filings.

Files

Usage

import joblib, pandas as pd
b = joblib.load('global_canonical.joblib')
X = df.reindex(columns=b['features'])          # canonical vocabulary
raw = sum(m.predict_proba(X)[:, 1] for m in b['members'].values()) / len(b['members'])
pd_calibrated = b['calibrator'].predict(raw)   # real probability of default

---

Generated 2026-08-31 from metrics files — numbers are not hand-entered.