Skip to content
sre-monitoring-and-observability logo

SRE Monitoring and Observability

sre-monitoring-and-observability

Use when building comprehensive monitoring and observability systems.

TheBushidoCollective/han0installs199stars

SKILL.md

Full skill instructions

SRE Monitoring and Observability

Building comprehensive monitoring and observability systems.

Four Golden Signals

Latency

Time to process requests:

# Request duration
http_request_duration_seconds

# Query
histogram_quantile(0.95, 
  rate(http_request_duration_seconds_bucket[5m])
)

Traffic

Demand on the system:

# Requests per second
rate(http_requests_total[5m])

# By endpoint
sum(rate(http_requests_total[5m])) by (endpoint)

Errors

Rate of failed requests:

# Error rate
rate(http_requests_total{status=~"5.."}[5m])
/ 
rate(http_requests_total[5m])

# SLI compliance
1 - (error_rate / slo_target)

Saturation

Resource utilization:

# CPU usage
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
/ node_memory_MemTotal_bytes * 100

Service Level Indicators (SLIs)

Availability SLI

# Successful requests / Total requests
sum(rate(http_requests_total{status=~"[23].."}[30d]))
/
sum(rate(http_requests_total[30d]))

Latency SLI

# Requests faster than threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[30d]))
/
sum(rate(http_request_duration_seconds_count[30d]))

Throughput SLI

# Requests processed within capacity
clamp_max(
  rate(http_requests_total[5m]) / capacity_requests_per_second,
  1.0
)

Alerting

Alert Severity Levels

P0 - Critical: Service down or severe degradation P1 - High: Significant impact, error budget at risk
P2 - Medium: Degradation, not user-facing yet P3 - Low: Awareness, no immediate action needed

Example Alerts

# High error rate
groups:
  - name: sre
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m])
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          
      - alert: LatencyP95High
        expr: |
          histogram_quantile(0.95,
            rate(http_request_duration_seconds_bucket[5m])
          ) > 1.0
        for: 10m
        labels:
          severity: warning
          
      - alert: ErrorBudgetBurn
        expr: |
          (1 - sli_availability) > (error_budget_remaining * 10)
        for: 1h
        labels:
          severity: high

Dashboards

Overview Dashboard

  • Service health (red/​yellow/​green)
  • Request rate
  • Error rate
  • Latency percentiles (p50, p95, p99)
  • Saturation metrics

Detailed Dashboard

  • Per-endpoint metrics
  • Dependency health
  • Database performance
  • Cache hit rates
  • Queue depths

Distributed Tracing

OpenTelemetry

const { trace } = require('@opentelemetry/​api');
const tracer = trace.getTracer('my-service');

async function handleRequest(req) {
  const span = tracer.startSpan('handle_request');
  
  try {
    span.setAttribute('user.id', req.user.id);
    span.setAttribute('request.path', req.path);
    
    const result = await processRequest(req);
    
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (error) {
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message,
    });
    throw error;
  } finally {
    span.end();
  }
}

Structured Logging

logger.info('request_processed', {
  request_id: req.id,
  user_id: req.user.id,
  endpoint: req.path,
  method: req.method,
  status_code: res.statusCode,
  duration_ms: duration,
  error: error?.message,
});

Best Practices

USE Method

For resources:

  • Utilization: % time resource is busy
  • Saturation: Work queued but not serviced
  • Errors: Error count

RED Method

For requests:

  • Rate: Requests per second
  • Errors: Failed requests per second
  • Duration: Request latency distribution

Alert on Symptoms, Not Causes

# Good - alert on user impact
- alert: HighLatency
  expr: p95_latency > 1s

# Bad - alert on potential cause
- alert: HighCPU
  expr: cpu_usage > 80%

Runbook Links

annotations:
  runbook: "https://wiki.example.com/​runbooks/​high-error-rate"
  dashboard: "https://grafana.example.com/​d/​abc123"

More skills from TheBushidoCollective

typescript-utility-types logo
TheBushidoCollective/han

typescript-utility-types

Use when typeScript utility types, mapped types, and advanced type manipulation. Use when creating flexible, type-safe TypeScript code.

199 0
View
shell-portability logo
TheBushidoCollective/han

shell-portability

Use when writing shell scripts that need to run across different systems, shells, or environments. Covers POSIX compatibility and platform differences.

199 0
View
tailwind-performance logo
TheBushidoCollective/han

tailwind-performance

Use when optimizing Tailwind CSS for production, reducing bundle size, and improving performance. Covers PurgeCSS, JIT mode, and build optimization.

199 0
View
figma-generate-component logo
TheBushidoCollective/han

figma-generate-component

Generate production-ready code from a Figma component or frame using the Figma MCP server

199 0
View
dialyzer-integration logo
TheBushidoCollective/han

dialyzer-integration

Use when integrating Dialyzer into development workflows and CI/CD pipelines for Erlang/Elixir projects.

199 0
View
rubocop-configuration logo
TheBushidoCollective/han

rubocop-configuration

Use when ruboCop configuration including YAML config, departments, inheritance, and customization.

199 0
View
elixir-otp-patterns logo
TheBushidoCollective/han

elixir-otp-patterns

Use when Elixir OTP patterns including GenServer, Supervisor, Agent, and Task. Use when building concurrent, fault-tolerant Elixir applications.

199 0
View
ameba-custom-rules logo
TheBushidoCollective/han

ameba-custom-rules

Use when creating custom Ameba rules for Crystal code analysis including rule development, AST traversal, issue reporting, and rule testing.

199 0
View
Swift Optionals Patterns logo
TheBushidoCollective/han

Swift Optionals Patterns

Use when swift's optional handling patterns including optional binding, chaining, nil coalescing, and modern approaches to safely working with optional values while avoiding common pitfalls and force unwrapping.

199 0
View
code-annotation-patterns logo
TheBushidoCollective/han

code-annotation-patterns

Use when annotating code with structured metadata, tags, and markers for AI-assisted development workflows. Covers annotation formats, semantic tags, and integration with development tools.

199 0
View
Scala Collections logo
TheBushidoCollective/han

Scala Collections

Use when scala collections including immutable/mutable variants, List, Vector, Set, Map operations, collection transformations, lazy evaluation with views, parallel collections, and custom collection builders for efficient data processing.

199 0
View
reactflow-custom-nodes logo
TheBushidoCollective/han

reactflow-custom-nodes

Use when creating custom React Flow nodes, edges, and handles. Covers custom node components, resizable nodes, toolbars, and advanced customization.

199 0
View

Popular AI tools

Kaiber logo
Video

Kaiber

Generate, edit, and beat-sync AI video with leading models in one workspace.

Paid
View
Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View

Transform Your Design with AI Designer by ImgCreator.ai

Freemium
View
Akool AI logo
Content & writing

Akool AI

Revolutionizing Video Production with AI-Powered Creativity

Paid
View

Extend an image past the frame and let AI fill the new aspect ratio.

Freemium
View
StarByFace logo
Security

StarByFace

Discover your celebrity doppelgänger with StarByFace!

Free
View
C

ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.

Freemium
View
Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View