Accuracy-Is-Not-Enough-FinQA-LoRA

FinQA Qwen2.5-7B LoRA

A LoRA adapter for structured financial question answering over Natively Extended FinQA.

Base model:

Qwen/Qwen2.5-7B-Instruct

This repository contains PEFT adapter weights, not a standalone copy of the base model.

Official Result

Selected checkpoint:

epoch_1_adapter

The checkpoint was selected using development-set performance before final test evaluation.

Intended Input

This adapter was trained on full expanded FinQA context.

Training mapping:

question
+ pre_text
+ table
+ post_text
        ↓
FinQA program

Training target:

qa.program

The selected prompt is included in this repository as:

S2_financial_analyst_operation_reader.json

Do not replace the full-document input with RAG chunks when attempting to reproduce the reported LoRA result.

Adapter Configuration

rank:        64
alpha:       32
dropout:     0.05
bias:        none
task type:   CAUSAL_LM

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Additional confirmed training settings include:

learning rate:         1e-4
effective batch size:  32
training dtype:        bfloat16

Official Training Burden

For the study's controlled practical comparison:

Training time: 14.7519 hours
Training cost: approximately $27.88

Installation

pip install torch transformers peft accelerate safetensors

Load the Adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-LoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Prepare the Input

Use the exact released S2_financial_analyst_operation_reader prompt.

The prompt should receive:

qa.question
pre_text
table
post_text

Do not expose:

qa.program
qa.exe_ans
qa.gold_inds

during normal dev or test inference.

Generate

Once rendered_prompt has been constructed with the official prompt:

inputs = tokenizer(
    rendered_prompt,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

generated = output[0, inputs["input_ids"].shape[-1]:]

text = tokenizer.decode(
    generated,
    skip_special_tokens=True,
)

print(text)

Expected Output

["subtract(", "5829", "5735", ")", "EOF"]

Evaluation

Parsed predictions should be converted to:

[
  {
    "id": "example-id",
    "predicted": [
      "subtract(",
      "5829",
      "5735",
      ")",
      "EOF"
    ]
  }
]

Then evaluate them with the original FinQA evaluator.

Related Repositories

Dataset:

Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-Dataset

Prompts, evaluator, results, and paper materials:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

Limitations

This adapter was trained specifically for FinQA-style numerical program generation.

It is not a general financial-advice model, and its reported accuracy should not be assumed to transfer directly to unrelated financial documents or tasks.

License

The adapter is released under the MIT License.

The Qwen base model remains subject to its own license.

The FinQA-derived training dataset is separately released under CC BY 4.0.