indian-nbfc-regulatory-assistant-v3

Indian NBFC Regulatory Assistant (v3)

A QLoRA fine-tune of Qwen/Qwen3-8B, adapted for question-answering over Reserve Bank of India (RBI) regulations governing Non-Banking Financial Companies (NBFCs) — lending conduct, digital lending, credit risk, governance, cybersecurity, reporting, and related compliance domains. Deployed as a live demo with a retrieval-confidence gate at huggingface.co/spaces/abhishek1995s/indian-nbfc-regulatory-assistant.

⚠️ This is a portfolio/research prototype, not a production compliance tool. It has NOT been validated for real-world regulatory decision-making. See Limitations before relying on any output. Always verify against rbi.org.in or a qualified compliance professional.

Last updated: 2026-08-23 · Deployed Space commit: 82146a0

---

What this model does

Given a question about RBI/NBFC lending regulation, this model — paired with a retrieval-confidence gate (see Gate Architecture) — either:

1. Answers with a citation to the specific RBI Direction/circular and section it's grounded in, or
2. Declines to answer when retrieval confidence is low across all gate signals, rather than guessing.

Intended use: portfolio/technical demonstration of an end-to-end fine-tuning + retrieval + safety-gating pipeline; a starting point for further work (per-class threshold tuning, larger/better-balanced training data, human-in-the-loop review).

Out-of-scope use: real compliance decision-making without human verification; any use where a fabricated citation could cause financial, legal, or regulatory harm; general-purpose assistant use (this is narrowly trained on RBI/NBFC lending regulation only).

---

Validated performance

Two separate validation passes, kept distinct because they measure different things:

Full 1,192-example labeled set (gate calibration)

The gate's two embedding-based thresholds and the nearest-neighbor bank's floor were calibrated against this set. Its "false-block" figure below should be read with one caveat: the nearest-neighbor bank was built from real failures observed within this same set, so its 0.1% figure reflects "known cases fixed," not pure generalization — the independent 100-question sample below is the fairer read on generalization.

Independent 100-question live test (real production endpoint, post-deploy)

100 fresh questions — none overlapping the calibration set or the nearest-neighbor bank — run against the live respond() endpoint. Full question/answer log: test_100_results.txt in the Space repo.

The 3 false-blocks are a known, documented, open issue — see Limitations.

---

How it was built

Fine-tuning

• Base model: Qwen/Qwen3-8B
• Method: QLoRA (4-bit quantized base for training, LoRA rank 16, alpha 32, attention projections only — q/k/v/o_proj)
• Trainable parameters: 15,335,424 (0.19% of the 8.2B total)
• Training data: 1,192 synthetic instruction pairs, generated via distillation from Qwen3.8-27B (Apache 2.0 licensed, distillation-permitted) against a curated corpus of 55 RBI/NBFC regulatory documents (3,858 retrieval chunks). Every generated answer passed an automated grounding check (does the answer's claims appear in the cited source chunk) before inclusion — ~78-85% of generated attempts passed; failures were discarded, not corrected.
• Class balance: training data intentionally oversampled decline/web_search examples (6x) relative to rag_local examples, after an earlier version showed the model rarely learned when not to answer.
• Training run: 3 epochs, ~1h on a single RTX 4090 (QLoRA), final eval_loss 0.9789.
• Corpus scope: NBFC-only (not banks or Housing Finance Companies), current-rules-only (superseded documents excluded from generation, tagged separately in the corpus for historical reference). Themes: lending conduct & disclosure, digital & product-specific lending, credit risk & prudential norms, governance/outsourcing/compliance, technology/cyber/fraud, reporting/audit/disclosure.

Production hardening (this deployment session)

Fine-tuning alone gets a model to reasonable answers; getting it to run reliably in production surfaced a separate set of infrastructure bugs, each diagnosed from live failures (not guessed) and fixed in order:

1. PEFT/bf16 dtype mismatch — PEFT loads adapter checkpoints in float32 regardless of the base model's dtype, causing "expected mat1 and mat2 to have the same dtype" crashes. Fixed by explicitly casting the wrapped model to bf16 after attaching the adapter.
2. OOM during model loading — device_map="auto" invoked a multi-threaded parallel tensor-materialization loader whose peak memory ran well above the model's final resident size. Fixed by loading onto CPU first, then moving to CUDA in one explicit call.
3. Concurrent model-load race — two simultaneous requests could each start loading their own full copy of the model, exceeding GPU memory even though either alone would fit. Fixed with a lock around the load path.
4. Decoding degeneration — an overly aggressive repetition penalty, combined with more generation headroom, forced the model into incoherent word-chain collapse on longer answers. Fixed by tuning repetition_penalty down and removing no_repeat_ngram_size.
5. Citation corruption — citation lines were garbled character-by-character (e.g. "ofIndia", "Directio ns, 2 025") even though the answer body decoded cleanly. Root cause, confirmed from the transformers source: no_repeat_ngram_size blocks n-grams across the entire prompt, not just newly-generated text — and the retrieved-chunk context repeats the same document-title phrase up to 19 times, so the model was blocked from correctly reproducing it in the citation. Fixed by removing that parameter.

Gate Architecture

Because fine-tuning alone doesn't reliably teach a model when to decline (see Limitations), a deterministic gate sits in front of this model — it only generates an answer if at least one of three signals clears its bar.

1. Gap-pattern regex detector (hard veto, checked first) — catches queries where the topic is well-covered by the corpus but the specific claim (e.g. "was X formally repealed, and on what exact date?") isn't confirmable, a failure mode retrieval-confidence alone can't detect since the topic still scores highly relevant.
2. Regulation-text embedding threshold (0.7991, chosen via Youden's J) — the query's embedding similarity against the corpus itself.
3. Evidence-grounded nearest-neighbor bank — a second embedding signal added specifically to fix casually-phrased-but-legitimate questions (e.g. "things to check while giving a personal loan") that don't lexically resemble regulatory prose and so score below the regulation-text threshold despite being fully answerable.

Signals 2 and 3 combine via OR logic (either passing is enough); signal 1 overrides both.

Honest note on how signal 3 was built: an earlier attempt used a large (2,880-question) generically-generated question bank as this second signal — it was rejected before deployment because it regressed the false-proceed rate on risky queries from 7.7% to 67.9% (generic "sounds like a compliance question" similarity doesn't distinguish in-scope topics from out-of-scope-but-similarly-phrased ones). The deployed version instead builds its bank from real observed gate failures: 263 confirmed cases the regulation-text gate incorrectly blocked, and 47 confirmed cases the rejected generic bank incorrectly let through — each expanded with 2 paraphrase variants. Its floor threshold was calibrated via Youden's J, and — because a bank built from known failures can trivially "pass" a test built from the same failures — it was verified on 5 held-out queries that appear in neither bank before being deployed.

---

Limitations

Infrastructure-level issues (fixed this session, 0/100 in live validation): dtype mismatches, OOM, generation degeneration, citation-text corruption. See Production hardening above.

Gate-level issue (open, not yet fixed): the nearest-neighbor bank occasionally false-blocks legitimate questions that are lexically adjacent to an out-of-scope acronym present in its negative bank — e.g. "What is the formula for CRAR?" gets pulled toward a negative-bank example about CRR (Cash Reserve Ratio, a different and out-of-scope ratio), because the positive bank has no CRAR-specific anchor (CRAR questions usually already passed the regulation-text gate before this fix, so they were never captured as "rescue" examples). Same pattern for LCR/SLR. Measured impact: 3/80 (3.75%) false-block rate in the 100-question live test, down from 23.7% before this fix, but not zero.

Model-level issues (predate this session, not addressed by infrastructure fixes — these are about factual/citation reliability, not text corruption):

• Fine-tuning alone did not reliably teach the model when to decline. Three independent training attempts (oversampling, heavier oversampling, explicit output-format tagging) each showed partial improvement but did not solve this — the base model's strong pretrained "be helpful" instinct appears to dominate a lightly-trained (0.19% of parameters) adapter. This is why the gate exists as a separate, deterministic layer in front of the model rather than relying on the model's own judgment.
• At least one confirmed case of cross-regulator citation fabrication in earlier evaluation — the model cited a SEBI (securities regulator) regulation when asked an RBI/NBFC question, with full confidence and a plausible-looking citation format.
• Citation format is inconsistent — roughly half of answers use the exact trained Source: ... format; the rest embed the citation naturally in prose without the literal prefix.

Given these findings, do not use this model's raw output for actual compliance decisions. It is deployed paired with a gate specifically because the model's own judgment about when to answer was found to be unreliable.

---

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype=torch.bfloat16)
base_model = base_model.to("cuda")
model = PeftModel.from_pretrained(base_model, "abhishek1995s/indian-nbfc-regulatory-assistant-v3")
model = model.to(torch.bfloat16)  # PEFT loads adapters in fp32 regardless of base dtype -- see Limitations
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

messages = [{"role": "user", "content": "Your RBI/NBFC lending question here"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=2000,
    repetition_penalty=1.15,
    do_sample=False,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Strongly recommended: pair with the retrieval-confidence gate (see Gate Architecture) rather than using this adapter's raw output directly — the base model's own judgment about when to decline was found to be unreliable in evaluation. Reference implementation: retrieval/gate.py in the Space repo.

---

License

Apache 2.0, matching the base model's license. Training data was generated via distillation from Qwen3.8-27B (Apache 2.0, distillation-permitted license terms).