Skip to content
performance-monitor logo

Performance Monitor

performance-monitor

Expert in observing, benchmarking, and optimizing AI agents. Specializes in token usage tracking, latency analysis, and quality evaluation metrics. Use when optimizing agent costs, measuring performance, or implementing evals. Triggers include "agent performance", "token usage", "latency optimiza...

NeverSight/skills_feed0installs215stars

SKILL.md

Full skill instructions

Performance Monitor

Purpose

Provides expertise in monitoring, benchmarking, and optimizing AI agent performance. Specializes in token usage tracking, latency analysis, cost optimization, and implementing quality evaluation metrics (evals) for AI systems.

When to Use

  • Tracking token usage and costs for AI agents
  • Measuring and optimizing agent latency
  • Implementing evaluation metrics (evals)
  • Benchmarking agent quality and accuracy
  • Optimizing agent cost efficiency
  • Building observability for AI pipelines
  • Analyzing agent conversation patterns
  • Setting up A/​B testing for agents

Quick Start

Invoke this skill when:

  • Optimizing AI agent costs and token usage
  • Measuring agent latency and performance
  • Implementing evaluation frameworks
  • Building observability for AI systems
  • Benchmarking agent quality

Do NOT invoke when:

  • General application performance → use /​performance-engineer
  • Infrastructure monitoring → use /​sre-engineer
  • ML model training optimization → use /​ml-engineer
  • Prompt design → use /​prompt-engineer

Decision Framework

Optimization Goal?
├── Cost Reduction
│   ├── Token usage → Prompt optimization
│   └── API calls → Caching, batching
├── Latency
│   ├── Time to first token → Streaming
│   └── Total response time → Model selection
├── Quality
│   ├── Accuracy → Evals with ground truth
│   └── Consistency → Multiple run analysis
└── Reliability
    └── Error rates, retry patterns

Core Workflows

1. Token Usage Tracking

  1. Instrument API calls to capture usage
  2. Track input vs output tokens separately
  3. Aggregate by agent, task, user
  4. Calculate costs per operation
  5. Build dashboards for visibility
  6. Set alerts for anomalous usage

2. Eval Framework Setup

  1. Define evaluation criteria
  2. Create test dataset with expected outputs
  3. Implement scoring functions
  4. Run automated eval pipeline
  5. Track scores over time
  6. Use for regression testing

3. Latency Optimization

  1. Measure baseline latency
  2. Identify bottlenecks (model, network, parsing)
  3. Implement streaming where applicable
  4. Optimize prompt length
  5. Consider model size tradeoffs
  6. Add caching for repeated queries

Best Practices

  • Track tokens separately from API call counts
  • Implement evals before optimizing
  • Use percentiles (p50, p95, p99) not averages for latency
  • Log prompt and response for debugging
  • Set cost budgets and alerts
  • Version prompts and track performance per version

Anti-Patterns

Anti-PatternProblemCorrect Approach
No token trackingSurprise costsInstrument all calls
Optimizing without evalsQuality regressionMeasure before optimizing
Average-only latencyHides tail latencyUse percentiles
No prompt versioningCan't correlate changesVersion and track
Ignoring cachingRepeated costsCache stable responses

More skills from NeverSight

devops-deployer logo
NeverSight/skills_feed

devops-deployer

Comprehensive DevOps and deployment workflow that orchestrates infrastructure automation, CI/CD pipelines, container orchestration, and cloud deployment. Handles everything from infrastructure as code and pipeline setup to monitoring, scaling, and disaster recovery.

215 1
View
avnu logo
NeverSight/skills_feed

avnu

Integrate avnu SDK for Starknet DeFi: token swaps with best-price routing, DCA recurring buys, native staking, gasless transactions via paymaster, and market data. Use when building Starknet trading, DEX aggregation, or DeFi automation features.

215 0
View
post-to-x logo
NeverSight/skills_feed

post-to-x

发布内容和文章到 X (Twitter)。支持带图片的普通推文和 X Articles(长文)。使用真实的 Chrome 和 CDP 来绕过自动化检测。

215 0
View
ralph-prompt-builder logo
NeverSight/skills_feed

ralph-prompt-builder

Master orchestrator for generating Ralph Wiggum-compatible prompts. Analyzes task requirements and routes to appropriate generator (single-task, multi-task, project, or research). Use when you need to create any Ralph loop prompt and want automatic selection of the right generator.

215 0
View
branch-comparison logo
NeverSight/skills_feed

branch-comparison

Use when the user asks to compare branches, analyze git diffs, review changes between branches, update specifications based on code changes, or analyze what changed. Triggers on keywords like "compare branches", "git diff", "what changed", "branch comparison", "code changes", "spec update".

215 0
View
fastapi-development logo
NeverSight/skills_feed

fastapi-development

Build high-performance FastAPI applications with async routes, validation, dependency injection, security, and automatic API documentation. Use when developing modern Python APIs with async support, automatic OpenAPI documentation, and high performance requirements.

215 0
View
zendesk-chat logo
NeverSight/skills_feed

zendesk-chat

Enables Claude to manage Zendesk Chat conversations, support tickets, and customer service operations

215 0
View
forge logo
NeverSight/skills_feed

forge

Mine transcripts for knowledge - decisions, learnings, failures, patterns. Triggers: "forge insights", "mine transcripts", "extract knowledge".

215 0
View
mixpanel logo
NeverSight/skills_feed

mixpanel

Track product analytics and user behavior with Mixpanel's event-based platform.

215 0
View
ocpp logo
NeverSight/skills_feed

ocpp

OCPP protocol reference for EV charging infrastructure development. Covers OCPP 2.0.1 and OCPP 1.6J. Use when working with OCPP messages, charging station code, CSMS/Central System backends, smart charging, transaction handling, or EV charging protocols. Activates on keywords: OCPP, charging stat...

215 0
View
bee-collect logo
NeverSight/skills_feed

bee-collect

Gather scattered ideas into organized GitHub issues. The bee collects pollen from every flower, checks if the hive already has it, and carefully deposits new work in the comb for later. Never edits code—only organizes the work ahead. Use when creating issues from TODOs or planning work.

215 0
View
vercel-react-native-skills logo
NeverSight/skills_feed

vercel-react-native-skills

Comprehensive best practices for React Native and Expo applications. Contains rules across multiple categories covering performance, animations, UI patterns, and platform-specific optimizations.

215 0
View

Popular AI tools

Kaiber logo
Video

Kaiber

Generate, edit, and beat-sync AI video with leading models in one workspace.

Paid
View
Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View

Transform Your Design with AI Designer by ImgCreator.ai

Freemium
View
Akool AI logo
Content & writing

Akool AI

Revolutionizing Video Production with AI-Powered Creativity

Paid
View

Extend an image past the frame and let AI fill the new aspect ratio.

Freemium
View
StarByFace logo
Security

StarByFace

Discover your celebrity doppelgänger with StarByFace!

Free
View
C

ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.

Freemium
View
Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View