finnai-slm-v2

FinnAI SLM v2

A 1.7B on-device language model for Indian bank SMS parsing and personal finance coaching.

FinnAI SLM is a Qwen/Qwen3-1.7B derivative fine-tuned with QLoRA SFT for the FinnDot expense tracker app. It runs entirely on-device — no cloud calls, no user data leaves the phone.

What it does

The model handles English, Hindi, Hinglish, Tamil, Telugu, Marathi, and Bengali bank SMS.

Key results (v2 held-out eval)

Test set: 1,098 non-empty transaction SMS + 184 empty/non-transaction SMS (OTP, promo, failed UPI, …). Ship gates still use the full mix; the extraction comparison below is non-empty rows only.

Non-empty rows only (1,098 real transactions)

On real spends, untuned Qwen3 already reads amounts well (~97%). FinnAI’s gain is mostly full-field match (merchant / account / type together).

Empty / non-transaction rows (184) — separate product check

Untuned chat models over-help: they emit JSON even when nothing is there. Fine-tuning teaches refusal. (Overall R-EM on all 1,282 rows: FinnAI 97.97%, Qwen3 28.16%.)

Chat groundedness (50-item set): FinnAI 68%, Qwen3 26%.

SHIP: YES. Gates 1–4 pass. McNemar p = 7.6e-270. Qwen3 column is the untuned base; FinnAI is the fine-tune. Non-empty metrics derived from published aggregates + the 1,098 / 184 split.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "finndot/finnai-slm-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

# SMS parsing
messages = [
    {"role": "system", "content": "Extract transaction details from this SMS as JSON: {amount, merchant, type, account, balance, category}. type is one of: INCOME, EXPENSE, CREDIT, TRANSFER, INVESTMENT. If this is not a transaction (OTP, promo, KYC), return {}."},
    {"role": "user", "content": "HDFC Bank: Rs.499.00 debited from A/c XX1234 to SWIGGY via UPI. Avl Bal Rs.15000.50"}
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
result = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(result)
# {"amount": 499.0, "merchant": "SWIGGY", "type": "EXPENSE", "account": "1234", "balance": 15000.5, "category": "Food"}

Training details

Data recipe (privacy-safe)

No user SMS. No real financial data. Fully synthetic.

Split hygiene: assignment unit is (bank, template_id), not individual SMS. All variants of a template stay in one split. See ml/data/manifest/SHA256SUMS.

Evaluation protocol

Pre-registered before looking at test scores: docs/llm-eval-protocol.md

Ship gates

1. SMS R-EM lift vs Qwen3-base ≥ +3 pp, CI excludes 0
2. Amount EM drop vs base ≤ 1 pp
3. Chat groundedness drop vs base ≤ 5 pp
4. JSON validity ≥ 95%
5. On-device TTFT / peak RSS ≤ 1.3× prior on-device baseline

Limitations

• Indian bank SMS only — trained on Indian banking formats (UPI, NEFT, IMPS, etc.). May not generalize to US/EU bank SMS.
• Merchant extraction is the hardest field; regex parsers in parser-core still win on supported banks.
• Chat groundedness is improved in v2 (~68%) but still not perfect; the model can paraphrase numbers.
• Thinking is disabled — do not expect chain-of-thought reasoning.
• 1.7B parameter model — smaller than general-purpose assistants; optimized for the SMS+coach task mix.
• Untuned bases over-parse — raw Qwen3 almost always invents a transaction on OTP/promo SMS; do not use the base model for this task without fine-tuning.

On-device deployment

The production format is LiteRT-LM INT4 (.litertlm, ~0.9 GB). The bf16 safetensors here are the merge before quantization. For the quantized on-device file, see the FinnDot app's model download.

Conversion: litert-torch export_hf, dynamic_int4_block32, KV 1280, nothink prefix.

Reproduce

git clone https://github.com/AnomaProAI/Finndot
cd Finndot/ml

# Build dataset
python -m data.build_splits --repo-root .. --out data/out --train-size 16000

# Train (needs 1× A10G / T4 GPU)
python train/sft_qlora.py --config train/train_config.yaml

# Merge adapter
python train/merge_lora.py --adapter <output>/adapter --out <output>/merged

# Evaluate
python -m eval.run_eval \
  --backend hf \
  --sms data/out/test_sms.jsonl \
  --chat eval/fixtures/chat_eval.jsonl \
  --model-id <output>/merged \
  --baseline-id Qwen/Qwen3-1.7B \
  --tag my-eval

Supported banks (35+)

HDFC, SBI, ICICI, Axis, Kotak, PNB, Bank of Baroda, Union Bank, IDFC First, Canara, Federal, Indian Bank, IOB, Central Bank, Karnataka, South Indian, City Union, DBS, HSBC, IDBI, J&K Bank, AMEX, OneCard, Slice, LazyPay, Juspay, Airtel Payments, Jio Payments, IPPB, Jupiter, Utkarsh, Mashreq, and more.

Citation

@misc{finnai-slm-2026,
  title={FinnAI SLM: On-device Indian bank SMS parsing with QLoRA fine-tuning},
  author={FinnDot Team},
  year={2026},
  url={https://huggingface.co/finndot/finnai-slm-v2},
  note={Apache 2.0. Based on Qwen3-1.7B.}
}

License

Apache 2.0 — same as the Qwen3-1.7B base model.