Datadog Monitoring

Datadog Monitoring

This skill provides access to Datadog for monitoring, logging, and alerting via the Datadog API.

Setup Required

You need to set up API credentials:

1. Go to Datadog → Organization Settings → API Keys
2. Create or copy an API Key
3. Go to Organization Settings → Application Keys
4. Create an Application Key

Set these as environment variables (add to your shell profile or .env):

export DD_API_KEY="your-api-key"
export DD_APP_KEY="your-application-key"
export DD_SITE="us3.datadoghq.com"  # Your Datadog site (from browser history: us3)

When to Use

Use this skill when the user:

• Asks about logs, errors, or application behavior
• Wants to check monitor/​alert status
• Needs to investigate an incident
• Asks about metrics or performance
• Mentions "Datadog" or monitoring

API Endpoints

Base URL: https://api.$(printenv DD_SITE)/​api/​v1 or v2

Logs

Search Logs (POST /​api/​v2/​logs/​events/​search):

curl -s -X POST "https://api.$(printenv DD_SITE)/​api/​v2/​logs/​events/​search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/​json" \
  -d '{
    "filter": {
      "query": "service:my-service status:error",
      "from": "now-1h",
      "to": "now"
    },
    "sort": "-timestamp",
    "page": {"limit": 50}
  }'

Common log query filters:

• service:name - Filter by service
• status:error - Filter by log level (error, warn, info, debug)
• @http.status_code:500 - Filter by HTTP status
• host:hostname - Filter by host
• env:production - Filter by environment

Monitors (Alerts)

List All Monitors (GET /​api/​v1/​monitor):

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​monitor" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Get Monitor by ID (GET /​api/​v1/​monitor/​{id}):

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​monitor/​{MONITOR_ID}" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Search Monitors:

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​monitor?query=status:Alert" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Metrics

Query Metrics (GET /​api/​v1/​query):

curl -s -G "https://api.$(printenv DD_SITE)/​api/​v1/​query" \
  --data-urlencode "query=avg:system.cpu.user{*}" \
  --data-urlencode "from=$(date -v-1H +%s)" \
  --data-urlencode "to=$(date +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

List Available Metrics (GET /​api/​v1/​metrics):

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​metrics?from=$(date -v-1d +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Events

Query Events (GET /​api/​v1/​events):

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​events?start=$(date -v-1d +%s)&end=$(date +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Dashboards

List Dashboards (GET /​api/​v1/​dashboard):

curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​dashboard" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Incidents

List Incidents (GET /​api/​v2/​incidents):

curl -s "https://api.$(printenv DD_SITE)/​api/​v2/​incidents" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Common Workflows

Check for Recent Errors

# Search for error logs in the last hour
curl -s -X POST "https://api.$(printenv DD_SITE)/​api/​v2/​logs/​events/​search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/​json" \
  -d '{
    "filter": {
      "query": "status:error",
      "from": "now-1h",
      "to": "now"
    },
    "page": {"limit": 25}
  }' | jq '.data[] | {timestamp: .attributes.timestamp, message: .attributes.message, service: .attributes.service}'

Check Alert Status

# List monitors that are currently alerting
curl -s "https://api.$(printenv DD_SITE)/​api/​v1/​monitor?query=status:Alert" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" | jq '.[] | {name, overall_state, message}'

Investigate a Service

# Get logs for a specific service
curl -s -X POST "https://api.$(printenv DD_SITE)/​api/​v2/​logs/​events/​search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/​json" \
  -d '{
    "filter": {
      "query": "service:SERVICE_NAME",
      "from": "now-30m",
      "to": "now"
    },
    "page": {"limit": 100}
  }'

Log Query Syntax

Datadog uses a powerful query syntax for logs:

Time Ranges

For the from and to parameters:

• now - Current time
• now-1h - 1 hour ago
• now-1d - 1 day ago
• now-7d - 1 week ago
• Unix timestamps (seconds)

Notes

• Your Datadog site appears to be us3.datadoghq.com based on browser history
• API rate limits apply - be mindful of query frequency
• Log queries return max 1000 results per request; use pagination for more
• Use jq to parse JSON responses
• Monitor status values: OK, Alert, Warn, No Data