llm-output-drift-financial-workflows

LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows

Authors: Raffi Khatchadourian, Rolando Franco
Venue: AI4F @ ACM ICAIF 2025 (Nov 15 in Singapore)

Paper: https://arxiv.org/abs/2511.07585
Code: https://github.com/ibm-client-engineering/output-drift-financial-llms

This repository is a Hugging Face landing page for the paper and its open-source implementation. It focuses on deterministic test harnesses, cross-provider validation, and risk-tiered deployment for financial LLM workflows (SEC 10-Ks, RAG over filings, JSON/SQL tasks).

🔑 Key finding

Well-engineered 7–8B models achieve 100% output consistency at T=0.0, while a 120B model reaches only 12.5% consistency, regardless of configuration.

Across 480 runs (5 models, 3 tasks, 2 temperatures, 3 concurrency levels), we show an inverse relationship between model size and determinism and map this to regulatory requirements (FSB, BIS, CFTC).

---

📊 Model tier classification

n = 480 runs (16 per condition), 95% Wilson CIs, p < 0.0001 (Fisher’s exact).

---

🎯 Why this matters

Financial institutions face a “verification tax”: human review erodes AI productivity gains when outputs are nondeterministic.

This framework shows:

• Audit-ready determinism is achievable with the right model + decoding setup.
• Cross-provider consistency: behavior transfers between local (Ollama) and cloud (IBM watsonx.ai).
• Task-specific drift: SQL and structured summaries remain stable even at T=0.2; RAG is far more sensitive.