qwen2.5-3b-dora-indo-transaction

Qwen2.5-3B DoRA — Indonesian Transaction Extraction

DoRA (Weight-Decomposed LoRA) adapter for Qwen/Qwen2.5-3B-Instruct, fine-tuned to extract structured JSON from Indonesian bank-SMS and e-wallet messages.

Task

Parse an Indonesian transaction notification into a fixed JSON schema:

Input:  "BCA 06/01 14:30 Transfer Rp1.500.000 ke BUDI SANTOSO Ref 8821 Sisa Rp3.200.000"
Output: {"bank":"BCA","type":"transfer","amount":1500000,
         "counterparty":"BUDI SANTOSO","datetime":"06/01 14:30",
         "balance":3200000,"ref":"8821"}

Fields: bank, type, amount, counterparty, datetime, balance, ref.

Results (400-example held-out test)

Training

• Method: DoRA (4-bit NF4 QLoRA + weight decomposition), PEFT + TRL SFTTrainer
• Data: 2,000 synthetic Indonesian transaction messages (8 transaction types, 8 banks + 5 e-wallets, varied number formats and date styles)
• Steps: 120 gradient steps, batch size 8, lr 2e-4
• Hardware: RTX 3060 12 GB

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base = "Qwen/Qwen2.5-3B-Instruct"
adapter = "fikri0o0/qwen2.5-3b-dora-indo-transaction"

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                          bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="cuda")
model = PeftModel.from_pretrained(model, adapter)
tok = AutoTokenizer.from_pretrained(adapter)

Part of the indo-transaction-extraction portfolio project by Muhammad Fikri Wahidin.