banking77-llama-1b-lora

banking77-llama-1b-lora

LoRA adapter for unsloth/Llama-3.2-1B-Instruct, fine-tuned on the mteb/banking77 77-class banking intent classification benchmark using QLoRA + rsLoRA + NEFTune.

90.21% exact-match accuracy on the Banking77 test split (2,775 / 3,076), 90.28% with trie-constrained decoding (2,777 / 3,076) Prompted zero-shot with the same instruction, the base model produced no correct labels (0 / 3,076) One free-tier T4 GPU · 3 epochs · 25 minutes of training (train_runtime 1,507 s)

---

Evaluation Results

All rows are greedy-decoded (do_sample=False) exact-match scores on the full Banking77 test split (3,076 queries, 77 classes, ~40 per class) from a single 3-epoch training run. Only the 9,993 train rows are written to the training data, so the model never trains on a test row. The run has no separate validation split: the test split is also the trainer's evaluation set, its loss by epoch was 0.4100, 0.3817 and 0.3839, and load_best_model_at_end on eval_loss kept the epoch-2 checkpoint, which is the model the fine-tuned rows score. During training, a generation check on a fixed 500-query random subset of the same split scored 80.80%, 89.00% and 90.60% after epochs 1, 2 and 3.

Exact match needs the precise snake_case label string. Given the same instruction, the base model never produced one, so the 90.2 pp gain covers both learning the output format and learning to tell 77 intents apart.

Constrained decoding builds a token-level prefix trie over all 77 valid label names and, at every decoding step, allows only tokens that continue a valid label, so every output is a valid label. On the full test split it fixed 2 samples and broke 0 (+0.07 pp). One fix replaced the invalid get_virtual_card with the correct get_disposable_virtual_card; in the other, the per-step restriction moved greedy decoding from one valid label (getting_spare_card) to the correct one (disposable_card_limits).

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama-3.2-1B-Instruct",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "rajo0113/banking77-llama-1b-lora")
model = model.merge_and_unload()  # optional: merge weights for faster inference
tokenizer = AutoTokenizer.from_pretrained("rajo0113/banking77-llama-1b-lora")

The reported figures were measured in the training notebook with the base model loaded in 4-bit NF4 through Unsloth (adapter_config.json names unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit). Loading the base in 16-bit as above can change a small number of predictions.

Inference

The model was trained with Llama's native chat template and the instruction below. Use both unchanged at inference:

INSTRUCTION = (
    "Classify the following banking customer query into one of 77 intent categories. "
    "Respond with only the intent label name."
)

def classify(query: str) -> str:
    messages = [{"role": "user", "content": f"{INSTRUCTION}\n\n{query}"}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=20, do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(
        out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip()

# Labels recorded for these queries by the notebook's inference test (Section 8):
print(classify("I lost my card, what should I do?"))
# → lost_or_stolen_card
print(classify("How do I change my PIN?"))
# → change_pin
print(classify("My transfer hasn't arrived yet."))
# → transfer_not_received_by_recipient

---

Training Configuration

Why the native chat template?

Using Llama's native chat format rather than Alpaca ### Instruction / ### Response was a central architectural decision. SFT builds on the format the model was already instruction-tuned with, instead of teaching it a new one, and the same template is used for training, evaluation and inference.

---

Dataset

mteb/banking77: 13,069 samples across 77 banking intent classes. Train split: 9,993 samples, the only rows written to the training data. Test split: 3,076 samples, used for evaluation and, as the trainer's only evaluation set, for checkpoint selection by eval loss. Labels are integer IDs (0-76) resolved to snake_case strings (e.g. card_arrival, transfer_not_received_by_recipient).

The task is framed as generative classification: the model outputs the label string directly, leveraging its text generation capability without requiring a classification head.

---

Worst-Class Analysis

Constrained decoding rules out the invalid top_up_by_card, but those 9 samples remain wrong: it cannot fix semantic confusion the model never learned to resolve during training. That points to the training data (for example, targeted topping_up_by_card examples) rather than model capacity.

---

Training Code & Full Methodology

Full training notebook (Google Colab T4) with the evaluation suite (per-class accuracy, confused pairs, zero-shot baseline, constrained decoding): github.com/rajo69/banking77-llama-1b-lora

---

Citation

@inproceedings{casanueva2020efficient,
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  author    = {Casanueva, I{\~n}igo and others},
  booktitle = {Proceedings of the 2nd Workshop on NLP for ConvAI},
  year      = {2020}
}
@article{hu2021lora,
  title  = {LoRA: Low-Rank Adaptation of Large Language Models},
  author = {Hu, Edward J and others},
  year   = {2021}
}
@article{dettmers2023qlora,
  title  = {QLoRA: Efficient Finetuning of Quantized LLMs},
  author = {Dettmers, Tim and others},
  year   = {2023}
}