agent-evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.
Full skill instructions
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
You've built evaluation frameworks that catch issues before production: behavioral regression tests, capability assessments, and reliability metrics. You understand that the goal isn't 100% test pass rate—it
Run tests multiple times and analyze result distributions
Define and test agent behavioral invariants
Actively try to break agent behavior
| Issue | Severity | Solution |
|---|---|---|
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
| Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
Works well with: multi-agent-orchestration, agent-communication, autonomous-agents
Accessibility guidelines for VS Code features — covers accessibility help dialogs, accessible views, verbosity settings, accessibility signals, ARIA alerts/status announcements, keyboard navigation, and ARIA labels/roles. Applies to both new interactive UI surfaces and updates to existing feature...
Mermaid diagram creation for flowcharts, sequences, ERDs, and more. Generate diagrams from text in markdown files. Use for documentation, architecture diagrams, and visual representations. Triggers on mermaid, flowchart, sequence diagram, ERD, entity relationship, gantt chart, pie chart, class di...
Guides development of cross-platform desktop apps with Tauri 2, TypeScript, React, and Rust. Use when building Tauri apps, implementing IPC, designing Rust backend or TypeScript/React frontend, when researching or cloning a website (open site, snapshot elements), when verifying local dev or built...
Handle errors as values using fp-ts Either and TaskEither for cleaner, more predictable TypeScript code. Use when implementing error handling patterns with fp-ts.
Gate 0 research phase for pre-dev workflow. Dispatches 3 parallel research agents to gather codebase patterns, external best practices, and framework documentation BEFORE creating PRD/TRD. Outputs research.md with file:line references.
Create and configure Vapi voice AI assistants with models, voices, transcribers, tools, hooks, and advanced settings. Use when building voice agents, phone bots, customer support assistants, or any conversational AI that handles phone or web calls.
Use when working with full stack orchestration full stack feature
Delegiere Aufgaben an OpenAI Codex CLI im Headless-Mode. Nutzt ChatGPT Subscription (KEIN API Key). Nur Workspace-Zugriff, KEIN System-Zugriff. Auto-Accept mit --full-auto. Web-Recherche mit --search.
Generate a full week of social media content for any topic. Outputs platform-optimized posts for Twitter/X, LinkedIn, and Instagram with hashtags and posting times.
Data privacy and regulatory compliance specialist for GDPR, CCPA, HIPAA, and international data protection laws. Use when implementing privacy controls, conducting data protection impact assessments, ensuring regulatory compliance, or managing data subject rights. Expert in consent management, da...
Reviews a task using the TRIPLE protocol, as the review agent.
Define and document the technology stack and architecture decisions for a project. Use when the user needs to choose a tech stack, make architecture decisions, define infrastructure choices, or document technology selections. Triggers on requests like "define the tech stack", "choose technologies...
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience
