sample-grpo-openenv

Fraud Detection RL Environment

An OpenEnv-compatible multi-agent RL environment for training and evaluating fraud detection policies.

Two agents interact every episode:

• Defender (LLM / PPO) — learns to detect and block fraudulent activity.
• Fraudster (LLM / PPO) — acts as an adaptive adversary trying to launder money.

The world is partially observable — agents see different, noisy views derived from the same hidden ground truth.

---

Quick start

# Install dependencies
pip install -e ".[dev]"
# or with uv:
uv sync

# Start the environment server
uv run server
# or: python -m scam_detection.server.app

# In a second terminal, run the LLM agent (inference)
export HF_TOKEN=<your_token>
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export API_BASE_URL=https://router.huggingface.co/v1
python inference.py

# Or run offline PPO training (no server needed)
python train.py --episodes 200 --task mule_cashout

# Evaluate trained vs baseline
python evaluate.py --episodes 20

With Docker:

docker build -t fraud-detection-env:latest .
docker run -p 8000:8000 fraud-detection-env:latest
python inference.py  # connects via ENV_URL

---

Fraud families (task variants)

---

Episode structure

reset(task_name=...)
  └─ FraudObservation:
       task_name, step_budget, fraud_family_hint
       defender_obs  (partial, noisy)
       fraudster_obs (partial, operational)
       available_defender_actions + targets
       available_fraudster_actions + targets

step(FraudAction(defender_action, fraudster_action, targets))   × N
  └─ FraudObservation:
       defender_obs / fraudster_obs (updated)
       defender_reward / fraudster_reward
       available_* actions
       info.grade (at terminal step)
       done, reason

Maximum steps per episode: 20 (configurable via DEFAULT_MAX_STEPS).

---

Action space

Defender actions

Fraudster actions

---

Reward structure

Defender per-step rewards

Fraudster per-step rewards

---

Observation contract

Agents never receive: ground-truth fraud labels, opponent's internal state, or raw reward equations.

# Reset observation
obs.task_name             # str — active fraud family
obs.step_budget           # {"total": 20, "used": 0, "remaining": 20}
obs.defender_obs          # dict — partial, noisy defender view
obs.fraudster_obs         # dict — partial, operational fraudster view
obs.available_defender_actions   # List[str]
obs.available_fraudster_actions  # List[str]
obs.defender_action_targets      # Dict[str, List[str]]
obs.fraudster_action_targets     # Dict[str, List[str]]

# Step observation (adds)
obs.defender_reward       # float
obs.fraudster_reward      # float
obs.info["grade"]         # dict — episode grade at terminal step
obs.done                  # bool
obs.reason                # "max_steps" | "all_routes_blocked" | "fraud_threshold_exceeded"

Defender observation fields

defender_obs = {
  "step": int,
  "fraud_family_hint": str,       # noisy hint (80% accurate)
  "accounts": [{
    "id", "transaction_velocity", "refund_ratio",
    "device_reuse_count", "risk_score",  # noisy
    "is_frozen", "is_challenged", "is_monitored",
    "balance_range",
  }],
  "merchants": [{
    "id", "transaction_count", "refund_rate",
    "anomaly_score",   # noisy
    "is_blocked", "is_monitored",
  }],
  "alerts": [{"type", "account_id"/"merchant_id", "step", ...}],
  "aggregate": {
    "total_frozen", "total_monitored", "total_flagged", "blocked_merchants",
    "recent_transaction_count",
  },
}

Fraudster observation fields

fraudster_obs = {
  "step": int,
  "alert_level": float,           # 0=safe, 1=detected
  "delayed_steps_remaining": int,
  "active_routes": [{
    "id", "family", "detection_pressure",
    "mule_status": [{"id", "is_frozen", "balance", "device_id"}],
    "merchant_blocked", "total_laundered", "cashout_ready", "steps_active",
  }],
  "available_mule_ids": List[str],
  "any_cashout_ready": bool,
  "total_laundered_so_far": float,
}

---

Episode grading metrics (terminal only)

Grading is saved to episode_grades.csv. Full rollout history (per-step observations, actions, rewards) is saved to rollout_history.csv for fine-tuning.

---

Project structure

scam_detection/
├── server/
│   ├── app.py                   # FastAPI server
│   └── fraud_environment.py     # FraudEnvironment (OpenEnv interface)
├── models.py                    # FraudAction, FraudObservation
├── constants.py                 # All constants
├── hidden_world_state.py        # Dataclasses for hidden world
├── scenario_generator.py        # ScenarioGenerator (world gen per family)
├── observation_generator.py     # ObservationGenerator (partial views)
├── action_processor.py          # ActionMasker + ActionProcessor
├── transition_engine.py         # TransitionEngine (world evolution)
├── reward_engine.py             # RewardEngine (per-step rewards)
├── termination_engine.py        # TerminationEngine (episode end)
├── grading_engine.py            # GradingEngine (episode metrics + CSV)
├── policy_networks.py           # DefenderPolicy + FraudsterPolicy (PPO)
├── ppo_trainer.py               # PPOTrainer (online training loop)
├── baseline_detector.py         # BaselineRuleDetector (non-RL baseline)
├── client.py                    # FraudEnvClient (typed async client)
├── inference.py                 # LLM agent runner (both agents)
├── train.py                     # PPO training script
├── evaluate.py                  # Evaluation + baseline comparison
├── tests/
│   └── test_smoke.py            # Smoke tests
├── pyproject.toml
├── Dockerfile
└── README.md

---

OpenEnv lifecycle

from scam_detection.client import FraudEnvClient
from scam_detection.models import FraudAction, DefenderActionType, FraudsterActionType

async with FraudEnvClient(base_url="http://localhost:8000") as env:
    # 1. Reset
    result = await env.reset(task_name="mule_cashout", seed=42)
    obs    = result.observation

    # 2. Steps
    for _ in range(20):
        action = FraudAction(
            defender_action=DefenderActionType.MONITOR,
            defender_target=obs.defender_action_targets.get("monitor", [None])[0],
            fraudster_action=FraudsterActionType.CASHOUT_ATTEMPT,
            fraudster_target=(obs.fraudster_action_targets.get("cashout_attempt") or [None])[0],
        )
        result = await env.step(action)
        obs    = result.observation
        if obs.done:
            print("Episode done:", obs.reason)
            print("Grade:", obs.info.get("grade"))
            break

---

Environment variables