IncidentTriage

OpenEnv-IncidentTriage

OpenEnv-IncidentTriage is a reinforcement-learning benchmark environment that simulates an on-call ML engineer responding to production incidents. An AI agent investigates synthetic logs, metrics, and alerts to diagnose issues, and then writes a diagnosis, fix plan, and postmortem — all within a specified step budget. It provides partial-credit rewards and features varied difficulties for evaluating agentic reasoning.

Architecture Diagram

+-------------------+       +---------------------+       +-----------------------+
|  Agent (LLM)      | <---> |  FastAPI Server     | <---> |  IncidentTriageEnv    |
|  (inference.py)   |       |  (app.py)           |       |  (environment.py)     |
+-------------------+       +---------------------+       |                       |
                            | - Rate Limiter      |       | - Step Budget         |
                            | - Secure Validation |       | - Tool Dispatch       |
                            | - State Management  |       | - Score Grader        |
                            +---------------------+       +-----------------------+
                                                                  |      |
                                        +-------------------------+      |
                                        |                                |
                                +-------v-------+                +-------v-------+
                                |  Tasks        |                |  Signals      |
                                | (task_easy,   |                | (JSON logs,   |
                                |  task_medium, |                |  metrics,     |
                                |  task_hard)   |                |  alerts)      |
                                +---------------+                +---------------+

Task Table

Action Space

Observation Space

Reward Function

The reward function relies on the quality of the submitted diagnosis, fix and postmortem, and penalises inefficiency. Total reward formula bounded in [0.0, 1.0]:

## Weights:
## diagnostic_accuracy = 0.40
## fix_correctness = 0.25
## postmortem_quality = 0.20
## step_efficiency = 0.15

raw = sum(weight * getattr(breakdown, key) for key, weight in REWARD_WEIGHTS.items())
penalty = max(-0.30, -0.01 * max(0, steps_taken - optimal_steps))
total = max(0.0, min(1.0, raw + penalty))

• Diagnosis Accuracy: NLP-based rubric to identify appropriate root causes and ignore red herrings.
• Fix Correctness: Check against required remediation actions and specific resolution commands.
• Postmortem Quality: Verifies presence of key document sections (Timeline, Root Cause, Impact, Remediation, Prevention, and Lessons Learned).
• Step Efficiency: Rewards resolving incidents within the optimal step window, degrading linearly down to 0 if max_steps are reached. Negative penalty for excessive steps is capped at -0.30.

Setup Instructions

1. Install locally:

   pip install -r requirements.txt
   

2. Docker Build:

   docker build -t openenv-incident-triage .
   

3. Docker Run:

   docker run -p 7860:7860 openenv-incident-triage
   

   The backend API will now be available on port 7860.

Running Inference

The inference.py provides the baseline LLM execution path solving all three tasks sequentially. Output adheres precisely to the standardized hackathon [START]/[STEP]/[END] format.

Environment Variables:

Example Command:

export HF_TOKEN="hf_yourtokenhere"
export ENV_BASE_URL="http://localhost:7860"
python inference.py

Validation

To evaluate environment compliance against OpenEnv specifications, ensure your backend server is deployed and run:

validate-submission.sh <your-hf-space-url>

You can also run validation using the openenv CLI tool if installed:

openenv validate

Running Tests

Verify environment integrity and high coverage with:

pytest tests/ -v --cov=openenv_incident_triage --cov-report=term

Security Notes

OpenEnv-IncidentTriage employs strict runtime boundaries to execute safely:

• No Remote Code Execution: Rejects eval/exec strings and prevents interpolation of inputs into OS-level commands or paths.
• Input Sanitization: Detects and denies shell injection syntax (;&|()), path traversal (../), null bytes, and excessively large payloads.
• Secrets Scanning: Heuristically warns about accidental leakage of API/HF/AWS tokens in agent submissions.
• Pydantic Validation: All inputs/outputs strictly typed; rejecting unknown configuration settings (extra="forbid").
• Deployment Hardening: Runs via non-root user appuser with a read-only signals/ store.

Contributing

You can expand this environment by authoring new scenarios:

1. Generate signal JSON records mapping simulated time to anomaly indicators in signals/.
2. Subclass BaseTask in a new file (e.g., tasks/task_custom.py):
   • Define signal_ids matching your new signal file IDs.
   • Implement logic for grade_diagnosis, grade_fix, and get_ground_truth.
3. Register your class in tasks/__init__.py.
4. Run pytest tests/ to confirm your grading criteria executes securely.

TO RUN $env:HF_TOKEN="your_working_token"; $env:ENV_BASE_URL="http://localhost:7860"; .\venv\Scripts\activate; python inference.py