Evals

Customization

Before executing, check for user customizations at: ~/​.claude/​skills/​CORE/​USER/​SKILLCUSTOMIZATIONS/​Evals/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:8888/​notify \
     -H "Content-Type: application/​json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \
     > /​dev/​null 2>&1 &
   

2. Output text notification:

   Running the **WorkflowName** workflow in the **Evals** skill to ACTION...
   

This is not optional. Execute this curl command immediately upon skill invocation.

Evals - AI Agent Evaluation Framework

Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).

Key differentiator: Evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.

---

When to Activate

• "run evals", "test this agent", "evaluate", "check quality", "benchmark"
• "regression test", "capability test"
• Compare agent behaviors across changes
• Validate agent workflows before deployment
• Verify ALGORITHM ISC rows
• Create new evaluation tasks from failures

---

Core Concepts

Three Grader Types

Evaluation Types

Key Metrics

• pass@k: Probability of at least 1 success in k trials (measures capability)
• pass^k: Probability all k trials succeed (measures consistency/​reliability)

---

Workflow Routing

---

Quick Reference

CLI Commands

# Run an eval suite
bun run ~/​.claude/​skills/​Evals/​Tools/​AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ~/​.claude/​skills/​Evals/​Tools/​FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ~/​.claude/​skills/​Evals/​Tools/​FailureToTask.ts convert-all

# Manage suites
bun run ~/​.claude/​skills/​Evals/​Tools/​SuiteManager.ts create <name> -t capability -d "description"
bun run ~/​.claude/​skills/​Evals/​Tools/​SuiteManager.ts list
bun run ~/​.claude/​skills/​Evals/​Tools/​SuiteManager.ts check-saturation <name>
bun run ~/​.claude/​skills/​Evals/​Tools/​SuiteManager.ts graduate <name>

ALGORITHM Integration

Evals is a verification method for THE ALGORITHM ISC rows:

# Run eval and update ISC row
bun run ~/​.claude/​skills/​Evals/​Tools/​AlgorithmBridge.ts -s regression-core -r 3 -u

ISC rows can specify eval verification:

| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |

---

Available Graders

Code-Based (Fast, Deterministic)

Model-Based (Nuanced)

---

Domain Patterns

Pre-configured grader stacks for common agent types:

See Data/​DomainPatterns.yaml for full configurations.

---

Task Schema (YAML)

task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/​security_review.md

  trials: 3
  pass_threshold: 0.75

---

Resource Index

---

Key Principles (from Anthropic)

1. Start with 20-50 real failures - Don't overthink, capture what actually broke
2. Unambiguous tasks - Two experts should reach identical verdicts
3. Balanced problem sets - Test both "should do" AND "should NOT do"
4. Grade outputs, not paths - Don't penalize valid creative solutions
5. Calibrate LLM judges - Against human expert judgment
6. Check transcripts regularly - Verify graders work correctly
7. Monitor saturation - Graduate to regression when hitting 95%+
8. Build infrastructure early - Evals shape how quickly you can adopt new models

---

Related

• ALGORITHM: Evals is a verification method
• Science: Evals implements scientific method
• Browser: For visual verification graders