finkpi-qwen3-0.6b

finkpi — structured KPI extraction from earnings press releases

A LoRA adapter over Qwen/Qwen3-0.6B that reads a passage of an earnings press release and emits the financial KPIs it states as JSON. Trained and evaluated on labels derived from filers' own XBRL submissions rather than from a larger model's guesses.

Results

Whole test split: 2890 passages from 31 companies that appear in no other split, all filed after every training filing.

Headline: micro F1 0.400 (95% CI 0.382–0.418) against a regex baseline of 0.356 (0.343–0.370). The intervals do not overlap.

The margin on F1 is real but modest. The larger differences are elsewhere:

This model is the more trustworthy reader, not the more complete one. The baseline finds more of what is there by firing constantly — on 0.455 of passages containing no KPI at all. Recall is the model's standing weakness and the obvious target for future work.

What fine-tuning actually bought

Prompting fixed the format completely and the arithmetic not at all. Shown three worked examples, the base model emits well-formed JSON with the right field names and still reads $12,973 under an (in millions) header as 1,297,300,000. Learning that a table cell inherits the document's scale header is what the fine-tune contributes, and it is why F1 moves from 0.000 to 0.400. Scale errors in the run above: 2 of 2233 predictions.

Where the errors are

metric_not_in_passage is the largest bucket, and a hand audit of a sample found most of them are real line items the label aligner did not capture — verbatim spans pointing at genuine figures the gold set is missing. So the precision above is an underestimate of the model's true precision. It is reported unadjusted because adjusting it would require judging the model's output by hand, which is exactly what this harness exists to avoid.

Per metric

Metrics with a score of 0.000 are not usable. total_debt has essentially no support in the corpus and should be treated as unsupported rather than poor.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(BASE)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(BASE, device_map="auto"), "REPO_ID"
).merge_and_unload().eval()

SYSTEM = open("prompt_system.txt").read()   # shipped in this repo; use verbatim
msgs = [{"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"Reporting period: Q3_2025\n\nPassage:\n{passage}"}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True)
out = model.generate(ids.to(model.device), max_new_tokens=768, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The prompt is part of the model. Change the system message and the numbers above no longer apply. max_new_tokens below ~768 truncates the longest answers: 2.2% of test passages need more than 320 tokens, and those hold 13.8% of all gold facts.

If you load in 4-bit, do not merge_and_unload(). Merging LoRA into NF4 weights means dequantise, add, requantise, and the requantisation error lands on exactly the weights the fine-tune changed.

Scope, and where it will fail

Trained on US-listed companies' English 8-K Item 2.02 exhibits. Expect degradation on non-US filers and IFRS presentations, segment and per-geography tables, guidance and forward-looking figures (excluded by construction), balance-sheet-heavy releases from financial institutions, and any metric outside the sixteen-item vocabulary.

Passage-level by design — it reads a window, not a whole release. For a document-level record, run every window and merge (finkpi.merge); agreement across overlapping windows is a usable confidence signal.

Not investment advice, and not a source of financial truth. It is a reading aid whose output should be checked against the filing. Every fact carries a verbatim span to make that check cheap; 0.972 of emitted spans are literal substrings of the input.

Training

QLoRA (4-bit NF4) on a single 4GB consumer GPU, peak 1.76 GB. 8806 examples, 1 epoch(s), rank 32, 4.6 hours. Prompt tokens masked; loss on the JSON answer only, and computed only at answer positions — full-sequence logits for this vocabulary are 594 MiB in fp32, larger than the quantised model, and do not fit.

Full configuration, losses and peak memory in train_metrics.json.

Reproducing

Everything, including the corpus build, is at the repository linked above. Per-run metrics for the baselines and this model are in eval_results/.

Provenance

{
  "git_commit": "",
  "python": "3.14.4",
  "platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
  "torch": "2.13.0+cu130",
  "transformers": "4.57.6",
  "peft": "0.20.0",
  "gpu": {
    "name": "NVIDIA GeForce RTX 3050 Laptop GPU",
    "total_gb": 4.0
  },
  "prompt_version": "1.1.0",
  "seed": 20260822
}