qwen3-1.7b-financial-qa-lora

Qwen3-1.7B Financial QA LoRA

This repository contains a LoRA adapter for Qwen/Qwen3-1.7B fine-tuned on normalized FinQA and DocFinQA financial question-answering examples.

This repository does not contain the base model weights. The adapter must be loaded on top of Qwen/Qwen3-1.7B.

Model details

• Base model: Qwen/Qwen3-1.7B
• Base model revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
• Adapter type: LoRA
• Training method: QLoRA with a 4-bit NF4 base model
• Language: English
• Domain: Financial question answering
• Task: Context-grounded numerical question answering

Intended use

The adapter is intended for:

• research on financial question answering;
• experiments with numerical reasoning over financial context;
• comparison between base and parameter-efficiently fine-tuned models;
• integration into retrieval-augmented generation prototypes.

The model is experimental and must not be treated as financial, investment, accounting, tax, or legal advice.

Training data

The adapter was trained on normalized examples from:

• FinQA;
• DocFinQA.

Each training example used:

1. a system instruction;
2. gold supporting context;
3. a financial question;
4. the reference final answer.

The training data itself is not included in this repository.

Dataset licenses

• FinQA: CC BY 4.0
• DocFinQA: MIT

Training configuration

Seven train examples, one validation example, and one final-holdout example were excluded because their rendered sequences exceeded 4,096 tokens (nine excluded examples in total).

Evaluation

Two evaluation sets were used:

• Development set: 100 examples used for comparison with smaller fine-tuning experiments.
• Final holdout: 1,937 eligible test examples not used for selecting the training subset size or hyperparameters.

Final holdout results

Numeric evaluation examples:

• Base model: 19 correct out of 1,902
• Fine-tuned adapter: 389 correct out of 1,902
• Improved examples: 372
• Regressed examples: 2

Numeric accuracy by dataset

Detailed aggregate results are available in the results/ directory.

Evaluation limitations

The reported numeric accuracy uses a deterministic custom evaluator that extracts and compares the first numeric value in each answer.

It is not the official FinQA program-execution or program-accuracy metric.

The evaluator does not fully normalize:

• percentages versus decimal ratios;
• thousands versus millions;
• currencies and units;
• multiple numeric values in one answer;
• semantically equivalent answers with different scales.

The results should therefore be reported as:

Numeric accuracy using the project evaluator v1.

Usage

Install the required libraries:

pip install \
  "transformers>=4.57.6" \
  "peft>=0.20.0" \
  "accelerate>=1.14.0" \
  "bitsandbytes>=0.50.0"

Load the base model and adapter:

import torch
from peft import PeftModel
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)

BASE_MODEL_ID = "Qwen/Qwen3-1.7B"
BASE_MODEL_REVISION = (
    "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
)
ADAPTER_ID = "fotapol/qwen3-1.7b-financial-qa-lora"

tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL_ID,
    revision=BASE_MODEL_REVISION,
)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID,
    revision=BASE_MODEL_REVISION,
    quantization_config=quantization_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_ID,
)

model.eval()

Generate an answer:

context = """
Revenue increased from $12.4 million in 2024 to
$14.1 million in 2025.
"""

question = """
What was the percentage increase in revenue?
"""

messages = [
    {
        "role": "system",
        "content": (
            "Answer the financial question using only the "
            "provided context. Return only the final answer."
        ),
    },
    {
        "role": "user",
        "content": (
            f"Context:\n{context}\n\n"
            f"Question:\n{question}"
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

answer = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(answer.strip())

Limitations

• The adapter was trained only on English data.
• It was trained with gold supporting context, not with retrieved context.
• It does not perform document retrieval.
• Numerical calculations can still be incorrect.
• DocFinQA performance remains lower than FinQA performance.
• The model may return plausible but unsupported financial statements.
• The training context was limited to 4,096 tokens.
• Evaluation used one custom metric implementation.
• The adapter has not been evaluated for financial advice or production decision-making.

Reproducibility

The repository includes:

• adapter weights and configuration;
• training configuration and metrics;
• experiment metadata;
• aggregate evaluation results;
• SHA-256 checksum for adapter weights.

The normalized training examples, split manifest, per-example predictions, full training notebook, and source pipeline are not included in this model release. Paths in the experiment metadata are relative to this repository; omitted artifacts are marked explicitly.

License

This adapter is released under the Apache License 2.0. See LICENSE for the complete license text. The Qwen3-1.7B base model is also distributed under Apache 2.0; the training datasets retain their respective licenses listed above.

Training software

Citation

FinQA

@inproceedings{chen2021finqa,
  title={FinQA: A Dataset of Numerical Reasoning over Financial Data},
  author={Chen, Zhiyu and others},
  booktitle={Proceedings of EMNLP},
  year={2021}
}

DocFinQA

@article{reddy2024docfinqa,
  title={DocFinQA: A Long-Context Financial Reasoning Dataset},
  author={Reddy, Varshini and others},
  year={2024}
}