Accuracy-Is-Not-Enough-FinQA-QLoRA

FinQA Qwen2.5-7B QLoRA

A QLoRA-trained PEFT adapter for structured financial question answering over Natively Extended FinQA.

Base model:

Qwen/Qwen2.5-7B-Instruct

QLoRA achieved the highest final-test accuracy of all six methods in the accompanying study.

Official Result

Selected checkpoint:

epoch_2_adapter

Epoch 2 was selected using development execution accuracy before final test evaluation.

Intended Input

This adapter was trained on full expanded FinQA context:

question
+ pre_text
+ table
+ post_text
        ↓
FinQA program

Training target:

qa.program

The selected prompt is included in this repository as:

S2_financial_analyst_operation_reader.json

QLoRA Training Configuration

Base-model quantization during training:

load_in_4bit:        true
quantization type:   NF4
compute dtype:       bfloat16
double quantization: true

PEFT adapter:

rank:        64
alpha:       32
dropout:     0.05
bias:        none
task type:   CAUSAL_LM

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Official Training Burden

Training time: approximately 19.28 hours
Training cost: approximately $36.43
Epochs completed: 3

QLoRA vs. Quantized Inference

QLoRA describes how the adapter was trained.

The released artifact is a PEFT adapter, not a standalone quantized copy of Qwen2.5-7B-Instruct.

The base model can be loaded in standard precision for inference or optionally loaded in a supported 4-bit configuration.

Standard Loading

pip install torch transformers peft accelerate safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-QLoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Optional 4-Bit Base-Model Loading

pip install bitsandbytes

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-QLoRA"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=quantization_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Prepare Input

Use:

qa.question
pre_text
table
post_text

through the official S2_financial_analyst_operation_reader template.

Do not expose:

qa.program
qa.exe_ans
qa.gold_inds

during normal validation or test inference.

Generate

After constructing the exact rendered_prompt:

inputs = tokenizer(
    rendered_prompt,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

generated = output[0, inputs["input_ids"].shape[-1]:]

print(
    tokenizer.decode(
        generated,
        skip_special_tokens=True,
    )
)

Expected Output

["divide(", "637", "const_5", ")", "EOF"]

Evaluation

Convert parsed output to the standard FinQA prediction structure and score it with the original FinQA evaluator.

Related Repositories

Dataset:

Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-Dataset

Research code, prompt, evaluator, results, and paper materials:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

Limitations

The 73.32% result is specific to the Natively Extended FinQA test set and the experimental configuration used in the accompanying study.

Broader financial-document performance requires separate evaluation.

License

The adapter is MIT licensed.

The base Qwen model remains under its upstream license.

The FinQA-derived training dataset is separately released under CC BY 4.0.