Agent Canvas

Agent Canvas

Interactive element picker that opens a browser window with a DevTools-like selection overlay. Users hover to highlight elements and click to select. Returns detailed element info including selector, bounding box, and computed styles.

First-Time Setup

Before first use, verify dependencies are installed:

uv run .claude/​skills/​agent-canvas-setup/​scripts/​check_setup.py check

If checks fail, ask user which installation scope they prefer and run:

# Recommended: minimal footprint, uv manages deps on-demand
uv run .claude/​skills/​agent-canvas-setup/​scripts/​check_setup.py install --scope temporary

# Alternative: create .venv in project
uv run .claude/​skills/​agent-canvas-setup/​scripts/​check_setup.py install --scope local

See agent-canvas-setup skill for full details on installation options.

Quick Start for AI Agents

When using agent-canvas, always follow this pattern:

1. Launch the picker (browser opens for user interaction)
2. Wait for browser to close (user finishes selecting/​editing)
3. Read session from disk (NOT from stdout - it may be lost)

# 1. Launch (user interacts with browser)
uv run .claude/​skills/​agent-canvas/​scripts/​agent_canvas.py pick http://localhost:3000 --with-edit --with-eyes

# 2. After browser closes, read the latest session
SESSION_ID=$(ls -t .canvas/​sessions/ | head -1)
cat .canvas/​sessions/​$SESSION_ID/​session.json | jq '.summary'

Commands

SKILL_DIR=".claude/​skills/​agent-canvas/​scripts"

Pick Element

Open browser with element picker overlay. Streams selection events as JSON lines until window is closed:

# Basic pick - opens browser, streams selections as JSON lines
uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000

# Pick with agent-eyes integration (adds screenshot + detailed styles per selection)
uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-eyes

# Pick with edit panel (floating DevTools for live style editing)
uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-edit

# Full workflow: picker + edit panel + agent-eyes (recommended)
uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-edit --with-eyes

# Save all selections and edits to file when done
uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-edit --output ./​session.json

User interaction:

1. Browser opens with blue highlight overlay
2. Hover over elements to see selector labels
3. Click to select (overlay flashes green, counter increments)
4. Keep clicking to select more elements
5. Close browser window when done

Streamed output (JSON lines):

{"event": "session_started", "url": "http://localhost:3000", "timestamp": "...", "features": {"picker": true, "eyes": true, "edit": true}}
{"event": "selection", "index": 1, "timestamp": "...", "element": {"tag": "button", "selector": "#submit", ...}}
{"event": "style_change", "timestamp": "...", "selector": "#submit", "property": "backgroundColor", "newValue": "#ff0000"}
{"event": "session_ended", "timestamp": "...", "total_selections": 1, "total_edits": 1}

With --with-eyes, each selection event also includes eyes (detailed styles) and screenshot fields. With --with-edit, style changes made in the floating panel are emitted as style_change events.

Watch for Changes

Monitor page for DOM changes, capture screenshots on each change:

# Watch with default 2s interval
uv run $SKILL_DIR/​agent_canvas.py watch http://localhost:3000

# Custom interval
uv run $SKILL_DIR/​agent_canvas.py watch http://localhost:3000 --interval 5

# Custom output directory
uv run $SKILL_DIR/​agent_canvas.py watch http://localhost:3000 --output-dir ./​snapshots

Outputs JSON events to stdout:

{"event": "watch_started", "url": "http://localhost:3000", "interval": 2.0}
{"event": "change_detected", "iteration": 1, "timestamp": "...", "screenshot": ".canvas/​screenshots/​..."}

Typical Workflow

1. Let user select element:

   uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-eyes
   

2. Use returned selector for further analysis:

   # Get accessibility info for selected element
   uv run .claude/​skills/​agent-eyes/​scripts/​agent_eyes.py a11y http://localhost:3000 --selector "#selected-element"
   

3. Make changes to the element's styles/​content

4. Verify changes with agent-eyes:

   uv run .claude/​skills/​agent-eyes/​scripts/​agent_eyes.py screenshot http://localhost:3000
   

Integration with Agent Eyes

When --with-eyes flag is used, agent-canvas calls agent-eyes to:

1. Get detailed element description (computed styles, attributes, visibility)
2. Take a screenshot of the selected element

This provides comprehensive visual context for the AI agent to understand and modify the selected element.

Integration with Canvas Edit

When --with-edit flag is used, agent-canvas loads the canvas-edit floating panel:

1. Users can visually adjust styles (colors, typography, spacing)
2. Changes apply live to the page for preview
3. Style change events stream alongside selection events
4. The panel uses Shadow DOM, so it's invisible to agent-eyes screenshots

Recommended workflow:

uv run $SKILL_DIR/​agent_canvas.py pick http://localhost:3000 --with-edit --with-eyes

This gives users full control to select elements, preview style changes, while the agent receives both the visual context and the specific CSS changes to implement.

Session Artifacts (IMPORTANT)

Sessions are automatically saved to .canvas/​sessions/<sessionId>/ regardless of how the command is run. This is the primary way to retrieve session data - do NOT rely on capturing stdout.

After Browser Closes - Read the Session

# List all sessions (most recent first)
ls -lt .canvas/​sessions/ | head -5

# Read the latest session
cat .canvas/​sessions/​$(ls -t .canvas/​sessions/ | head -1)/​session.json

# Or use jq for formatted output
cat .canvas/​sessions/​$(ls -t .canvas/​sessions/ | head -1)/​session.json | jq '.summary'

Session Structure

.canvas/​sessions/<sessionId>/
├── session.json    # Full event log, selections, edits, screenshots (base64)
└── changes.json    # Extracted save_request (if user clicked "Save All to Code")

Key Fields in session.json

{
  "sessionId": "ses-abc123",
  "url": "http://localhost:3000",
  "summary": {
    "totalSelections": 5,
    "totalEdits": 3,
    "hasSaveRequest": true  // <-- Check this! false = user didn't save changes
  },
  "events": {
    "selections": [...],  // Element selection events with screenshots
    "edits": [...]        // Style/​text changes from edit panel
  }
}

Checking What Changed

# Quick summary
cat .canvas/​sessions/<sessionId>/​session.json | jq '.summary'

# See all selections (element info)
cat .canvas/​sessions/<sessionId>/​session.json | jq '.events.selections[] | {selector: .payload.element.selector, text: .payload.element.text}'

# See all edits
cat .canvas/​sessions/<sessionId>/​session.json | jq '.events.edits'

# Check if save was requested (required for canvas-apply)
cat .canvas/​sessions/<sessionId>/​session.json | jq '.summary.hasSaveRequest'

Notes

• Browser launches in visible mode for pick command (user interaction required)
• Browser runs headless for watch command
• Selection events stream as JSON lines to stdout in real-time
• Session artifacts are always saved to disk - use these instead of stdout capture
• Close the browser window to end the session
• Overlay elements are excluded from selection