rinlekha-gemma3-4b-finetuned

RinLekha — NBFC Credit Memo Generator

Fine-tuned Gemma 3 4B IT (QLoRA, rank 16) on 640 synthetic credit memo examples for Indian NBFCs. Given a structured borrower profile, the model produces a fully-formatted 6-section credit memo with an APPROVE / CONDITIONAL APPROVE / DECLINE recommendation.

Model Details

Intended Use

• In-scope: Generating draft credit assessment memos for personal/MSME loan applications at Indian NBFCs, following a defined institutional format.
• Out-of-scope: Final lending decisions, regulatory compliance, markets outside India, loan amounts beyond the training distribution.

The model is a research prototype. All outputs require human review before use in any real lending workflow.

Training Data

Dataset: a-anurag1024/rinlekha-training-data

• 800 synthetic borrower profiles generated via stratified sampling (employment type × income band × CIBIL tier × FOIR tier)
• Decision labels assigned by a deterministic rule engine matching real NBFC underwriting policy (55% FOIR ceiling, 620 CIBIL floor, settled-account thresholds)
• Memos synthesised with GPT-4.1-mini; QC pass filters hallucinations and format violations
• Final split: 640 train / 80 validation / 80 test (80/10/10)

Training Procedure

Three-run ablation over rank and learning rate. Best run selected on validation loss:

Training environment: Unsloth + TRL on Google Colab T4 (15 GB VRAM). LoRA adapter merged via PEFT merge_and_unload() and exported to GGUF Q8_0 via llama.cpp.

Evaluation Results — 100 Test Cases

Results by decision type:

Evaluation harness: 6 custom DeepEval metrics (3 rule-based, 1 count-based, 2 LLM-judge via GPT-4o-mini). Results logged to MLflow.

Baseline Comparison — RinLekha vs GPT-4o-mini (30 cases)

GPT-4o-mini is the cost-comparable API baseline — similar inference cost tier to a self-hosted 4B GGUF. Both models receive the same format instructions in the system prompt; any gap on structural metrics reflects format internalization from fine-tuning rather than prompt engineering.

The most striking gap is RiskFlagsCount (0.967 vs 0.133): GPT-4o-mini consistently produces fewer than 2 or more than 4 bulleted risk flags even with explicit instructions. This is the kind of tight structural constraint that fine-tuning internalizes reliably while prompting does not. GEval (analytical quality) is essentially tied, confirming the 4B model matches a much larger prompted model on reasoning quality for this domain-specific task.

Adversarial Evaluation — 8 Edge Cases

Hand-crafted cases targeting boundary conditions not well-represented in the test set. Decision accuracy: 62% (5/8) — all DECLINE cases correct, misses on CONDITIONAL APPROVE.

Adversarial aggregate metrics (format compliance holds even on edge cases): StructuralCompliance 1.000 · RecommendationFormat 1.000 · ForbiddenLanguage 1.000 · RiskFlagsCount 1.000 · GEval 0.853 · Faithfulness 0.959

Failure pattern: the model correctly identifies hard DECLINE signals (FOIR ceiling, settled accounts, delinquency) but conflates CONDITIONAL APPROVE with APPROVE on soft-boundary cases — particularly short employment tenure and unverifiable self-employed income. The training set under-represents these borderline conditions relative to clean approvals.

Usage

from llama_cpp import Llama

llm = Llama(model_path="rinlekha-q8.gguf", n_gpu_layers=99, n_ctx=2048)

INSTRUCTION = (
    "You are a senior credit analyst at an Indian NBFC. "
    "Write a structured credit memo for the borrower profile "
    "below following institutional format exactly."
)

prompt = f"### Instruction:\n{INSTRUCTION}\n\n### Input:\n{borrower_profile}\n\n### Response:\n"
out = llm(prompt, max_tokens=700, temperature=0.1, stop=["### Instruction:"])
print(out["choices"][0]["text"])

Or via the OpenAI-compatible server:

# Download GGUF
huggingface-cli download a-anurag1024/rinlekha-gguf rinlekha-q8.gguf --local-dir outputs/

# Start server (llama-cpp-python[server])
python -m llama_cpp.server --model outputs/rinlekha-q8.gguf --host 0.0.0.0 --port 8000 \
  --model_alias rinlekha --n_gpu_layers 99 --n_ctx 2048

Expected Output Format

## APPLICANT SUMMARY
...

## DEBT SERVICEABILITY
...

## CREDIT BEHAVIOR
...

## RISK FLAGS
- Risk flag 1
- Risk flag 2

## RECOMMENDATION
DECISION: CONDITIONAL APPROVE
CONDITIONS: ...
RISK GRADE: B
DECISION AUTHORITY: Branch Credit Manager
REVIEW TRIGGER: ...

## ANALYST NOTES
...

Limitations

• Training data is fully synthetic (GPT-4.1-mini authored). The model has not seen real loan files.
• The 55% FOIR ceiling and 620 CIBIL floor are hard-coded into the training labels; policy changes are not reflected without retraining.
• Max context is 2048 tokens. Very long borrower profiles may be truncated.
• The model occasionally generates mild hedged hallucinations in the Analyst Notes section; Faithfulness score of 0.961 reflects this.
• Evaluated at temperature 0.1 (near-greedy); higher temperatures degrade format compliance.

Repository

github.com/a-anurag1024/rinlekha