llama-3-8b-finance-json

Llama-3-8B Finance JSON Extractor - Model Card

A QLoRA fine-tune of Llama-3-8B that converts unstructured financial news into a fixed JSON record: ticker, sentiment_score, event_type, impact_summary.

Read this before quoting any number

• The hold-out set has 19 examples. Confidence intervals on every metric below are wide. Differences of a few points between systems are within noise and should not be reported as a ranking.

Evaluation results

Hold-out set: data/test_data.jsonl, n=19, produced by src/data/split_dataset.py (seed 42) and verified free of train/test overlap by src/data/validate_dataset.py. Decoding is greedy (temperature 0), so these numbers are reproducible.

How to read the base-model column. The base model gets the same prompt but has never seen the four field names, so it invents its own JSON layout (typically a generic entities list). Its low scores mostly show that fine-tuning taught the model this schema. They are not evidence that the fine-tune beats a well-prompted instruction-tuned model; that comparison has not been run.

macro-F1 averages per-class F1 over the event types present in the test set, so a model that only predicts the majority class scores poorly. This is deliberately not the composite score used by the earlier version of this project, which was a hand-weighted average of accuracies and was mislabelled as F1. See src/eval/metrics.py.

Coverage gap: 1 of the 10 declared event types has zero examples in the hold-out set, so macro-F1 is averaged over 9 classes only. The model's behaviour on the missing classes is unmeasured.

Per-class breakdown (fine-tuned)

Throughput, latency and cost

Throughput, latency and cost: not yet measured. Run src/bench/benchmark.py against a live vLLM deployment. Until then this model card makes no performance or cost claim.

Training

Training run metadata: not yet measured (results/run_metadata.json absent).

The published adapter predates this harness, so its wall-clock time, peak VRAM and final loss were never recorded. The LoRA geometry below is read directly from the published adapter_config.json and is therefore verifiable; the rest is not.

Re-run python src/train.py to produce a fully documented run.

Training data

100 unique labelled snippets, split 81/19 train/test (seed 42). 5 duplicate rows and 0 schema-invalid rows were removed before splitting.

Test-set support by class:

Labels were generated by GPT-4o acting as a teacher model and validated against the schema. They are not human-annotated and not market-derived. sentiment_score in particular is one model's judgement, so agreement with it is distillation fidelity, not correctness.

Licence and attribution

This adapter is a derivative of Meta Llama 3 8B and is governed by the Meta Llama 3 Community License, not Apache-2.0. Redistribution must include the notice "Built with Meta Llama 3" and a copy of the licence, and the Llama 3 acceptable-use policy applies. Verify the current terms at https://llama.meta.com/llama3/license/ before publishing.

Intended use and limits

• Structured extraction from short English financial news snippets.
• Not investment advice and not a trading signal. sentiment_score is a language-model judgement of tone, not a return forecast.
• Evaluated only on the small in-domain hold-out set described above. Behaviour on longer documents, non-US tickers, other languages and adversarial input is unmeasured.

---

Generated by scripts/render_model_card.py from the files in results/. Do not hand-edit; regenerate after each measurement run.