openenv-secaudit

SecretsAuditEnv

A reinforcement-learning benchmark that drops an AI agent into a git-backed codebase seeded with realistic secret leaks — hardcoded API keys, base64-encoded tokens, credentials buried in git history — and grades how quickly and safely the agent remediates every one. The environment runs as a stateless FastAPI server; the agent interacts over HTTP with bash commands and structured inspection actions, receiving a composite reward after every step that captures security progress, code health, and time efficiency.

Why This Exists

Production secret leaks remain a top-5 cause of cloud breaches. Developers commit API keys, push .env files, or leave credentials in migration scripts — then scramble to rotate and rewrite history. Existing linting tools flag secrets but don't fix them; LLM agents can, but there's no standardized benchmark to measure how well. SecretsAuditEnv fills that gap with a 13-task curriculum spanning trivial single-file fixes to multi-service cascading leaks with git-history rewriting, complete with a deterministic grading pipeline and an anti-gaming reward function.

---

---

Task Curriculum

All 13 tasks are defined in tasks/ with full metadata in each task.json. Secret types are detected by graders/security.py using regex patterns for AWS keys, GitHub tokens, Firebase keys, connection strings, private keys, SQL passwords, assignment secrets, and base64-encoded variants.

Scan modes: dir scans the working directory only. git scans the full git commit history (all revisions), so agents must use git filter-repo to clean history — deleting files won't work.

---

Secret Visibility Tiers

Secrets are not all visible at episode start. The environment implements a 4-tier progressive disclosure system defined per-secret in task.json:

Hard tasks (11–13) include conflict maps that encode dependencies: fixing secret s1 may reveal s3, while s2 may block s3 until resolved.

---

Reward Formula

Defined in graders/reward.py. Computed fresh after every /step:

detection_score = (initial_leaks - current_leaks) / initial_leaks
base            = 0.4 × detection_score + 0.6 × detection_score     # (remediation = detection for now)
health_score    = pytest_passed / pytest_total                       # 0.0 if errors > 0 or total == 0
efficiency      = 0.15 × max(0, 1 - steps_taken / step_budget)      # decays linearly per step

if base > 0:
    total_reward = min(1.0, base × health_score + efficiency)
else:
    total_reward = 0.0                                               # no free points for doing nothing

Key properties:

• Reward is 0.0 until the agent actually fixes something — read-only commands like cat or ls cannot earn reward
• Health gate: breaking the test suite (e.g., deleting imports) multiplies reward toward zero
• Efficiency bonus (0.0–0.15): rewards agents that solve in fewer steps. Decays to 0 at step budget
• Step budgets: Easy = 10, Medium = 20, Hard = 30

---

Action Space

Actions are sent as the action field in POST /step. Two categories:

Structured Actions (intercepted before bash)

Bash Commands (executed in workspace via /usr/bin/bash -lc)

Any string that doesn't match a structured prefix is executed as a bash command in the task workspace. Common patterns:

• cat config.py — read file contents
• sed -i 's/AKIA.../os.getenv("AWS_KEY")/' config.py — redact a secret
• git filter-repo --replace-text <(echo 'ghp_xxx==>REDACTED') --force — clean git history
• gitleaks detect --no-git --source . — run leak scanner

Commands time out after 90 seconds. Exit code, stdout, and stderr are returned in the observation.

---

Observation Keys

Every /step and /reset response returns a session object with these fields (also listed in openenv.yaml):

Ranked Actions

Generated by server/observation.py using 5 heuristics:

1. Visible unfixed secrets → priority 0.92 (fix these first)
2. Uninspected high-risk files → priority based on filename suspicion score
3. Git history not yet scanned → priority 0.75 (medium/hard only)
4. Hidden secrets remaining → priority 0.68 (suggest encoded inspection)
5. All visible fixed → priority 0.85 (run gitleaks validation)

---

Quick Start

Local Development

# Install dependencies
pip install -r requirements.txt

# Generate task workspaces
python tools/generate_tasks.py

# Start the server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Open the debug UI
open http://localhost:7860/web

# Run the agent (requires LLM API)
export API_BASE_URL="https://openrouter.ai/api/v1"
export HF_TOKEN="your-api-key"
export MODEL_NAME="nvidia/nemotron-3-super-120b-a12b:free"
export ENV_URL="http://localhost:7860"
python inference.py --task-id 1

Docker

docker build -t secretsauditenv .
docker run -p 7860:7860 secretsauditenv
# Server auto-generates tasks and starts on port 7860

The Dockerfile installs Python 3.11-slim with bash, git, git-filter-repo, and all pip dependencies. The entrypoint runs tools/generate_tasks.py then starts uvicorn.

---

inference.py — Agent Loop

The baseline agent (inference.py) implements a single-turn ReAct loop over any OpenAI-compatible API:

1. Reset — POST /reset with the target task ID
2. Prompt — Builds a structured prompt from the current session state (reward, leaks, health, last command output, recent actions)
3. Call LLM — Sends prompt to the model, extracts a bash command from the response
4. Normalize — Strips markdown fences, extracts from XML tool_call formats, filters prose prefixes, enforces atomic (no && or ; chaining)
5. Step — POST /step with the action, reads updated state
6. Repeat until reward >= 1.0 or MAX_STEPS (40) reached

Environment variables:

Anti-loop features:

• Detects 3 consecutive identical actions with identical rewards → injects a CRITICAL WARNING into the prompt and forces a different command
• Filters natural language preambles (e.g., "Let me check...", "Looking at...") before sending to bash

---

Grading Pipeline

Security Grader (graders/security.py)

Custom regex-based scanner (no external tools required). Detects:

• AWS Access Keys (AKIA...)
• GitHub tokens (ghp_...)
• Firebase API keys (AIza...)
• Service tokens (tok_live_..., sk_test_...)
• Private keys (PEM format)
• SQL connection strings (postgres://user:pass@host/db)
• SQL passwords (PASSWORD 'value')
• High-entropy assignment secrets (Shannon entropy ≥ 3.2)
• Base64-encoded variants of all the above

Supports .gitleaks.toml allowlists. For scan_mode: git, scans all commits via git rev-list --all.

Anti-gaming: if an agent deletes .git, the grader recovers by creating a fresh snapshot and scanning that — the secret still gets found.

Health Grader (graders/health.py)

Runs pytest -q --junitxml in the task workspace and parses the JUnit XML report. Score = passed / total. Returns 0.0 if any errors or if pytest times out (60s default).

---

Web Debug UI

Available at GET /web. A single-page dark-mode dashboard served from server/web_ui.html that lets you:

• Start any of the 13 tasks with one click
• Send structured actions or raw bash commands
• View real-time metrics (reward, leaks, hidden count, steps, efficiency, health)
• See ranked action suggestions and conflict maps
• Browse visible secrets and full observation text
• Review action history

No external dependencies — pure HTML/CSS/JS.

---

Validation

# Run the environment test suite (34 tests)
python -m pytest tests/ -v

# Run the submission validator (checks connection, Docker build, spec compliance, reward integrity)
bash validate-submission.sh

The validator checks:

1. Connection — /reset returns HTTP 200
2. Docker portability — no absolute host paths leak into the image
3. Spec compliance — spec.md exists, all 13 task directories present, openenv validate passes
4. Reward integrity — smoke test runs inference.py for 2 steps and verifies [START]/[STEP]/[END] log format

---

Project Structure

.
├── server/
│   ├── app.py              # FastAPI routes (/reset, /step, /state, /tasks, /web)
│   ├── environment.py      # Core environment logic, visibility tiers, action parsing
│   ├── observation.py      # Ranked action heuristics and top_blocker computation
│   └── web_ui.html         # Debug dashboard (served at /web)
├── graders/
│   ├── security.py         # Regex-based secret scanner with git history support
│   ├── health.py           # Pytest-based health grader with JUnit parsing
│   ├── reward.py           # Composite reward: security × health + efficiency
│   ├── gitleaks_eval.py    # Spec-aligned wrapper with git integrity reporting
│   └── health_eval.py      # Spec-aligned wrapper with failure messaging
├── tasks/
│   ├── easy/task_01..05/   # 5 easy tasks (single-file, surface+shallow secrets)
│   ├── medium/task_06..10/ # 5 medium tasks (multi-format, surface+shallow+deep)
│   └── hard/task_11..13/   # 3 hard tasks (cascading, conflict maps, git history)
├── tests/
│   ├── test_hidden_leaks.py    # 11 tests: visibility tier logic
│   ├── test_ranked_actions.py  # 12 tests: heuristic action suggestions
│   └── test_reward.py         # 11 tests: reward formula and efficiency bonus
├── tools/
│   ├── generate_tasks.py   # Generates all 13 task workspaces with seeded secrets
│   ├── check_space_ready.py
│   ├── prepare_hf_space_bundle.sh
│   └── prepare_github_upload_bundle.sh
├── inference.py            # Baseline LLM agent loop (OpenAI-compatible)
├── openenv.yaml            # OpenEnv spec declaration
├── Dockerfile              # Python 3.11-slim + git + git-filter-repo
├── hf_space_entrypoint.sh  # Docker entrypoint: generate tasks → start uvicorn
├── validate-submission.sh  # 4-check submission validator
├── requirements.txt        # Pip dependencies
├── pyproject.toml          # Package metadata
└── spec.md                 # Environment specification document

---

License

MIT