Multi-App-RL-Env-Invoice-Processing-Schema-Drift-Fraud-Detection-Vendor-Negotiation

An RL Environment for Multi-App Enterprise Workflows: Schema Drift, Fraud Detection, and Vendor Negotiation Meta PyTorch OpenEnv Hackathon Grand Finale 2026 - Team: SHIPWithTEA

The Problem: Enterprise Accounts Payable (AP) departments process hundreds of invoices every day. The work looks repetitive on the surface — read the invoice, check it against the Purchase Order, approve or reject - but the reality is far more complicated. Vendors send incorrect amounts. ERP systems change their APIs without warning. Duplicate invoices slip through undetected. And increasingly, attackers send fraudulent invoices from lookalike domains that differ from the real vendor by a single character.

Current rule-based automation systems catch obvious errors but fail at anything that requires judgment. We wanted to know whether a language model could be trained to handle this entire workflow — and get measurably better at it over time.

The Environment We built the Enterprise AP Environment, an OpenEnv-compatible reinforcement learning environment that places an agent inside a simulated AP department. Each episode is a fresh, randomized scenario generated from a pool of five vendors, with unique invoice numbers, line items, quantities, prices, tax rates, and dates. The agent cannot memorize answers , every reset produces a genuinely new situation.

The agent interacts with three simulated systems:

An email inbox, where invoices arrive as unstructured text from vendor billing addresses An ERP system, which the agent must query using the correct API schema to retrieve the matching Purchase Order A vendor communication channel, where the agent can send and receive emails to resolve disputes The environment exposes a standard Gym-style API — reset, step, state — and is hosted as a REST API on Hugging Face Spaces, making it accessible to any HTTP client.

The Tasks The environment includes five tasks of increasing difficulty:

Easy — A clean, well-formatted invoice arrives. The agent must read it, query the ERP, extract seven structured fields (vendor name, invoice number, invoice date, due date, subtotal, tax amount, total), and approve.

Medium — The invoice line item prices are inflated compared to the Purchase Order. The agent must detect the mismatch, raise a flag, and reject.

Hard — The ERP API has silently changed its contract. The previous endpoint accepted vendor_name; the new one requires vendor_tax_id. The agent must recover from the schema error, find the tax ID in the invoice body, retry the query, and additionally detect that the invoice is a duplicate of one already processed.

Expert Negotiation — The invoice total exceeds the approved Purchase Order amount because the vendor forgot to apply the agreed partnership discount. The agent must email the vendor, wait for a corrected invoice to arrive in its inbox, read the corrected version, re-extract all fields, and approve.

Expert Fraud — The invoice itself is legitimate in content, but it was sent from a lookalike domain — for example, techsuppIies.com with a capital I instead of a lowercase l. The agent must scrutinize the sender address, flag it as fraud, and reject without being misled by the otherwise valid invoice data.

The Reward Design: The reward function is composable and operates at two levels.

At the step level, the agent receives small immediate rewards for correct actions: reading an email (+0.05), a successful ERP query (+0.10), correctly extracting a field (+0.07), raising the right flag (+0.12), and successfully negotiating with a vendor (+0.20). Wrong actions carry small penalties to discourage random exploration.

At the episode level, the final score is computed as a weighted sum across three components:

Field extraction accuracy across all seven fields — 40 percent Correct flags raised or negotiation completed — 30 percent Correct final decision (approve or reject) — 30 percent A hard floor of 0.10 is applied to any episode where the agent never queries the ERP system, enforcing the multi-app workflow. An agent that reads the email and approves without cross-referencing the Purchase Order will not score above 0.10 regardless of how accurate its extracted fields are.

Training Results We trained a rule-based agent with a linearly decaying noise schedule across all five tasks, running 60 episodes per task. The noise parameter simulates an agent that starts with a high error rate and progressively improves its decision quality — analogous to what gradient-based training produces in a language model.

Final average rewards across tasks:

Task Final Average Reward Target Result Easy 0.940 0.85 Pass Medium 0.940 0.75 Pass Hard 0.940 0.65 Pass Expert Negotiation 0.940 0.70 Pass Expert Fraud 0.940 0.70 Pass The Expert Negotiation task showed the most interesting learning trajectory. The agent begins at approximately 0.52 average reward because it frequently attempts to approve the overpriced invoice without first negotiating. By episode 48, it consistently executes the correct three-phase sequence: send negotiation email, read the corrected reply, re-extract fields, then approve. The reward curve for this task has the steepest early slope and the most visible inflection point.

What Makes This Environment Useful for Training Most RL environments for language models test a single capability — math reasoning, code generation, or factual retrieval. This environment tests a compound skill: multi-step, multi-app reasoning under adversarial conditions.

An agent that scores well on all five tasks has learned to:

Navigate between unstructured text (email) and structured data (ERP) Recover gracefully from API errors it has never seen before Distinguish legitimate invoices from fraudulent ones using only subtle signals Engage in a multi-turn negotiation and update its internal state based on the reply The domain is deliberately underexplored in LLM training research. Unlike chess or grid worlds, there is no fixed optimal sequence — every episode is different, the reward is dense and informative, and the failure modes (approving a fraudulent invoice, missing a duplicate, failing to negotiate) are the exact failures that cost real organizations real money.

Links Live environment: https://huggingface.co/spaces/decent-cow26/invoice-env

Source code and training script: https://github.com/dharmendra26-wiz/Enterprise-AP-Environment

Built for the Meta PyTorch OpenEnv Hackathon Grand Finale 2026 by Prachi and Dharmendra.