Skip to content
investigating-pagerduty-incidents logo

Workflow

investigating-pagerduty-incidents

Generate weekly PagerDuty incident reports for fan-audio and fusion services with root cause analysis from GCP logs. Outputs a Word document ready to paste into Slack.

SKILL.md

Full skill instructions

You are a PagerDuty incident investigator for the Music Mission MIQ team at Spotify.

Services: fan-audio, fusion Output: Word document with actionable insights and debugging links

Workflow

  1. Fetch incidents from past week (resolved + open)
  2. For fan-audio: Query GCP logs at incident timestamps to find root causes
  3. Generate concise Word document with key findings and action items

Commands

Fetch Incidents

pd incident list --since "$(date -v-7d '+%Y-%m-%d')" --statuses resolved --services "fan-audio" "fusion" --output json
pd incident list --services "fan-audio" "fusion" --output json

Get Alert Links (only for representative incidents, not all)

pd rest get -e /​incidents/​INCIDENT_ID/​alerts | jq '.alerts[0].body.contexts[0].href'

Query GCP Logs (fan-audio only)

For each unique error pattern, query a 1-hour window around incident time:

# Find actual errors at incident timestamp
gcloud logging read 'resource.labels.container_name="fan-audio-backend-worker" AND severity="ERROR" AND timestamp>="2025-11-19T22:00:00Z" AND timestamp<="2025-11-19T23:00:00Z"' \
  --project=fan-audio-2 \
  --limit=30 \
  --format=json | \
  jq '[.[] | select(.jsonPayload.exc_info or (.jsonPayload.message and (.jsonPayload.message | test("DEADLINE|TypeError|Exception") and (test("UserWarning|Clipped") | not))))] | .[0]'

# Get error frequency for the week
gcloud logging read 'resource.labels.container_name="fan-audio-backend-worker" AND severity="ERROR"' \
  --project=fan-audio-2 \
  --limit=200 \
  --freshness=7d \
  --format=json | \
  jq -r '[.[] | select(.jsonPayload.message and (.jsonPayload.message | test("Error|Exception|DEADLINE"))) | .jsonPayload.message] | group_by(.) | map({error: .[0], count: length}) | sort_by(.count) | reverse | .[:5]'

Generate Word Document

Create pagerduty-report-YYYY-MM-DD.docx using python-docx. Keep it concise and actionable.

Structure:

  1. Summary (3-4 bullets): Total incidents, services affected, urgency
  2. Key Issues (grouped by root cause, not individual incidents):
    • Pattern description
    • Actual error from logs (DEADLINE_EXCEEDED, TypeError, etc.)
    • Frequency/​impact
    • One representative debugging link
  3. Action Items (2-4 max): Concrete next steps based on root causes
  4. Links Section: Key debugging URLs as actual hyperlinks

Python Template:

from docx import Document
from docx.shared import RGBColor
from docx.oxml import OxmlElement
from docx.oxml.ns import qn

def add_hyperlink(paragraph, url, text):
    """Add a hyperlink to a paragraph."""
    part = paragraph.part
    r_id = part.relate_to(url, 'http://schemas.openxmlformats.org/​officeDocument/​2006/​relationships/​hyperlink', is_external=True)
    hyperlink = OxmlElement('w:hyperlink')
    hyperlink.set(qn('r:id'), r_id)
    new_run = OxmlElement('w:r')
    rPr = OxmlElement('w:rPr')
    new_run.append(rPr)
    new_run.text = text
    hyperlink.append(new_run)
    paragraph._p.append(hyperlink)
    return hyperlink

doc = Document()

# Title and date
doc.add_heading('PagerDuty Weekly Report', level=1)
p = doc.add_paragraph('Nov 13-20, 2025')
p.runs[0].italic = True

# Summary
doc.add_heading('Summary', level=2)
doc.add_paragraph('7 incidents (all resolved, low urgency)', style='List Bullet')
doc.add_paragraph('fan-audio: 6 incidents, fusion: 1 incident', style='List Bullet')

# Key Issues (group by root cause)
doc.add_heading('Key Issues', level=2)

p = doc.add_paragraph()
p.add_run('Spotbot gRPC Timeouts (fan-audio-backend-worker)').bold = True
doc.add_paragraph('51 DEADLINE_EXCEEDED errors when calling Spotbot for track recommendations', style='List Bullet')
doc.add_paragraph('Impact: 6 incidents over the week', style='List Bullet')
p = doc.add_paragraph('Link: ', style='List Bullet')
add_hyperlink(p, 'https://oliver.spotify.net/​ui/​component/​fan-audio-backend-worker/​incident', 'Oliver Investigation')

# Action Items
doc.add_heading('Action Items', level=2)
doc.add_paragraph('Add circuit breaker/​retry logic for Spotbot gRPC calls', style='List Bullet')
doc.add_paragraph('Add millisecond jitter to prevent session creation race conditions', style='List Bullet')

# Links
doc.add_heading('Debugging Links', level=2)
p = doc.add_paragraph()
add_hyperlink(p, 'https://console.cloud.google.com/​logs/​query;query=resource.labels.container_name%3D%22fan-audio-backend-worker%22%0Aseverity%3D%22ERROR%22;duration=P7D?organizationId=642708779950&project=fan-audio-2', 'GCP Logs - Backend Worker')

doc.save('pagerduty-report-2025-11-20.docx')

Best Practices

  • Group incidents by root cause, don't list every incident individually
  • Skip noisy details: Only include actionable error messages
  • Filter GCP logs: Ignore UserWarnings, focus on real errors (DEADLINE_EXCEEDED, TypeError, Exceptions)
  • Use actual hyperlinks: The add_hyperlink function makes clickable links that work when pasted into Slack
  • Be concise: 1 page max, focus on "what to do" not "what happened"
  • Run commands in parallel when possible (multiple Bash tool calls in one message)

After Generating Report

Tell the user:

  1. File path
  2. "Open in Word/​Pages, select all, copy, paste into Slack - links will work!"
  3. One-sentence summary of key finding

More Productivity & Planning skills

brainstorming logo
Productivity & Planning

brainstorming

Structured design dialogue that validates ideas before implementation begins.

295.4K 385.7K
View
ui-ux-pro-max logo
Productivity & Planning

ui-ux-pro-max

Comprehensive design intelligence for web and mobile UI/UX across 10 technology stacks.

133.1K 383.9K
View
writing-plans logo
Productivity & Planning

writing-plans

Comprehensive implementation plans for multi-step tasks, breaking down specs into bite-sized, testable steps.

295.4K 268K
View
using-superpowers logo
Productivity & Planning

using-superpowers

Introduction to the obra skills system with mandatory skill invocation rules and best practices.

295.4K 259.8K
View
executing-plans logo
Productivity & Planning

executing-plans

Execute a written implementation plan with critical review and task checkpoints.

295.4K 229.1K
View
dispatching-parallel-agents logo
Productivity & Planning

dispatching-parallel-agents

Delegate independent tasks to specialized agents working concurrently with isolated context.

295.4K 206.3K
View
using-git-worktrees logo
Productivity & Planning

using-git-worktrees

Isolated git worktrees with smart directory selection and safety verification.

295.4K 205.2K
View
webapp-testing logo
Productivity & Planning

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

179.7K 170.6K
View
content-strategy logo
Productivity & Planning

content-strategy

Plan searchable and shareable content that drives traffic, builds authority, and generates leads.

53.3K 150.9K
View
repo-intake-and-plan logo
Productivity & Planning

repo-intake-and-plan

README-first repository scanner that extracts commands and classifies reproduction candidates without executing them.

497 139.6K
View
marketing-ideas logo
Productivity & Planning

marketing-ideas

Brainstorm and prioritize marketing strategies tailored to your SaaS stage, budget, and goals.

53.3K 137.2K
View
site-architecture logo
Productivity & Planning

site-architecture

Plan and optimize your website's page hierarchy, navigation, URL structure, and internal linking.

53.3K 112.8K
View

Productivity AI tools

Vimcal logo
Productivity

Vimcal

The world's fastest calendar for remote work

Free
View
SaveDay logo
Productivity

SaveDay

Capture, organize, and utilize your knowledge effortlessly.

Free
View
Any Summary logo
Productivity

Any Summary

Instant Summaries of Audio & Video Interviews with AnySummary

Freemium
View
M
Productivity

Map This

Transform PDFs into engaging mind maps.

Freemium
View
ChatPDF logo
Productivity

ChatPDF

Chat with any PDF instantly

Free
View
I
Productivity

intellisay

Create an optimal daily plan using your voice

Paid
View
A
Productivity

Aurora AI

A productivity platform to centralize organizational knowledge and workflows with contextual AI assistance.

Paid
View
AskYourPDF logo
Productivity

AskYourPDF

AskYourPDF Pricing Plans: Tailored to Your Needs

Free
View