MaintainX Incident Runbook

MaintainX Incident Runbook

Overview

Step-by-step procedures for responding to MaintainX integration incidents, from detection through resolution.

Prerequisites

• Access to monitoring dashboards
• MaintainX admin credentials
• On-call contact information

Incident Classification

Incident Response Flow

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   DETECT    │────▶│   TRIAGE    │────▶│  DIAGNOSE   │
│             │     │             │     │             │
│ Alert fires │     │ Determine   │     │ Identify    │
│ User report │     │ severity    │     │ root cause  │
└─────────────┘     └──────┬──────┘     └──────┬──────┘
                          │                    │
                          ▼                    ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   CLOSE     │◀────│   RECOVER   │◀────│  MITIGATE   │
│             │     │             │     │             │
│ Post-mortem │     │ Verify fix  │     │ Stop impact │
│ Document    │     │ Monitor     │     │ Temporary   │
└─────────────┘     └─────────────┘     └─────────────┘

Runbook Procedures

RB-001: MaintainX API Unreachable

Symptoms:

• All API calls failing with network errors
• Health check endpoint returning unhealthy
• Timeout errors in logs

Diagnostic Steps:

#!/​bin/​bash
# scripts/​diagnose-api-unreachable.sh

echo "=== MaintainX API Connectivity Diagnostic ==="
echo ""

# Step 1: Basic connectivity
echo "1. Testing DNS resolution..."
nslookup api.getmaintainx.com

echo ""
echo "2. Testing TCP connectivity..."
nc -zv api.getmaintainx.com 443

echo ""
echo "3. Testing HTTPS endpoint..."
curl -v --max-time 10 https://api.getmaintainx.com/​v1/​users?limit=1 \
  -H "Authorization: Bearer $MAINTAINX_API_KEY" 2>&1

echo ""
echo "4. Checking MaintainX status page..."
curl -s https://status.getmaintainx.com/​api/​v2/​status.json | jq '.status.indicator'

echo ""
echo "5. Checking from different regions (if available)..."
# Add region-specific tests if you have multi-region deployment

Mitigation:

1. Check MaintainX Status Page
2. If MaintainX is down: Enable fallback mode / cached responses
3. If local network: Check firewall, DNS, proxy settings
4. Escalate to network team if needed

Resolution:

• Wait for MaintainX restoration
• Verify connectivity restored
• Monitor for data sync catch-up

---

RB-002: High Error Rate (>5%)

Symptoms:

• MaintainXHighErrorRate alert firing
• 4xx or 5xx errors in logs
• Failed work order operations

Diagnostic Steps:

// scripts/​diagnose-errors.ts

async function diagnoseHighErrorRate() {
  console.log('=== Error Rate Diagnostic ===\n');

  // Get recent error logs
  const errors = await getRecentErrors(100);

  // Group by error type
  const errorGroups = new Map<string, number>();
  errors.forEach(e => {
    const key = `${e.status}:${e.endpoint}`;
    errorGroups.set(key, (errorGroups.get(key) || 0) + 1);
  });

  console.log('Error distribution:');
  errorGroups.forEach((count, key) => {
    console.log(`  ${key}: ${count}`);
  });

  // Check for patterns
  console.log('\nCommon patterns:');

  const status401 = errors.filter(e => e.status === 401);
  if (status401.length > 0) {
    console.log('  - 401 Unauthorized: Check API key validity');
  }

  const status429 = errors.filter(e => e.status === 429);
  if (status429.length > 0) {
    console.log('  - 429 Rate Limited: Reduce request rate');
  }

  const status5xx = errors.filter(e => e.status >= 500);
  if (status5xx.length > 0) {
    console.log('  - 5xx Server Errors: MaintainX side issue');
  }
}

Mitigation by Error Type:

Resolution:

• Fix underlying issue
• Clear error state
• Monitor for recurrence

---

RB-003: Webhook Processing Failures

Symptoms:

• Webhooks not being processed
• Events missing in local system
• webhook_failures metric elevated

Diagnostic Steps:

#!/​bin/​bash
# Check webhook endpoint health
curl -v https://your-app.com/​webhooks/​maintainx/​health

# Check webhook queue depth
redis-cli LLEN maintainx:webhook:queue

# Check recent failed webhooks
redis-cli LRANGE maintainx:webhook:failed 0 10

# Test webhook signature verification
./​scripts/​test-webhook.sh

Mitigation:

1. Check webhook secret is correct
2. Verify endpoint is accessible from internet
3. Check for queue backlog
4. Temporarily pause webhook processing if overwhelmed

Resolution:

• Fix webhook handler
• Replay failed webhooks
• Verify event sync

---

RB-004: Data Sync Out of Sync

Symptoms:

• Local data doesn't match MaintainX
• Missing work orders or updates
• Stale data being served

Diagnostic Steps:

// scripts/​diagnose-sync.ts

async function diagnoseSyncIssues() {
  const client = new MaintainXClient();

  console.log('=== Sync Diagnostic ===\n');

  // Get sample from MaintainX
  const maintainxWOs = await client.getWorkOrders({ limit: 100 });

  // Get sample from local
  const localWOs = await localStore.getWorkOrders({ limit: 100 });

  // Compare
  const maintainxIds = new Set(maintainxWOs.workOrders.map(wo => wo.id));
  const localIds = new Set(localWOs.map(wo => wo.id));

  const missingLocally = [...maintainxIds].filter(id => !localIds.has(id));
  const extraLocally = [...localIds].filter(id => !maintainxIds.has(id));

  console.log(`MaintainX work orders: ${maintainxIds.size}`);
  console.log(`Local work orders: ${localIds.size}`);
  console.log(`Missing locally: ${missingLocally.length}`);
  console.log(`Extra locally: ${extraLocally.length}`);

  if (missingLocally.length > 0) {
    console.log('\nMissing IDs (first 10):');
    missingLocally.slice(0, 10).forEach(id => console.log(`  - ${id}`));
  }
}

Mitigation:

1. Trigger full sync if data is stale
2. Check webhook delivery
3. Verify polling job is running

Resolution:

• Run reconciliation script
• Fix sync process
• Monitor for drift

---

RB-005: Performance Degradation

Symptoms:

• MaintainXHighLatency alert firing
• Slow user experience
• Timeout errors

Diagnostic Steps:

// scripts/​diagnose-performance.ts

async function diagnosePerformance() {
  console.log('=== Performance Diagnostic ===\n');

  // Test API latency
  const endpoints = ['/​workorders', '/​assets', '/​locations', '/​users'];

  for (const endpoint of endpoints) {
    const start = Date.now();
    await client.request(`GET ${endpoint}?limit=10`);
    const latency = Date.now() - start;

    const status = latency > 1000 ? '[SLOW]' : '[OK]';
    console.log(`${status} ${endpoint}: ${latency}ms`);
  }

  // Check cache performance
  const cacheStats = await cache.getStats();
  console.log('\nCache stats:');
  console.log(`  Hit rate: ${cacheStats.hitRate}%`);
  console.log(`  Size: ${cacheStats.size}`);

  // Check connection pool
  const poolStats = await getConnectionPoolStats();
  console.log('\nConnection pool:');
  console.log(`  Active: ${poolStats.active}`);
  console.log(`  Idle: ${poolStats.idle}`);
  console.log(`  Waiting: ${poolStats.waiting}`);
}

Mitigation:

1. Enable aggressive caching
2. Reduce concurrent requests
3. Add request queuing

Resolution:

• Identify slow endpoints
• Optimize queries
• Scale resources if needed

Emergency Contacts

Post-Incident Template

## Incident Report: [TITLE]

**Date:** YYYY-MM-DD
**Duration:** HH:MM - HH:MM (X hours)
**Severity:** SEVX
**Status:** Resolved

### Summary
Brief description of what happened.

### Timeline
- HH:MM - Alert fired
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Mitigation applied
- HH:MM - Resolution confirmed

### Root Cause
What caused the incident.

### Impact
- X work orders affected
- Y minutes of downtime
- Z users impacted

### Resolution
What was done to fix it.

### Action Items
- [ ] Prevent recurrence
- [ ] Improve detection
- [ ] Update runbooks

Output

• Incident diagnosed
• Mitigation applied
• Root cause identified
• Resolution documented

Resources

• MaintainX Status Page
• MaintainX Support
• MaintainX API Documentation

Next Steps

For data handling patterns, see maintainx-data-handling.