dd-apm
APM - traces, services, dependencies, performance analysis.
SKILL.md
Full skill instructions
Datadog APM
Distributed tracing, service maps, and performance analysis.
Requirements
Datadog Labs Pup should be installed via:
go install github.com/datadog-labs/pup@latest
Quick Start
pup auth login
pup apm services list
pup apm traces list --service api-gateway --duration 1h
Services
List Services
pup apm services list
pup apm services list --env production
Service Details
pup apm services get api-gateway --json
Service Map
# View dependencies
pup apm service-map --service api-gateway --json
Traces
Search Traces
# By service
pup apm traces list --service api-gateway --duration 1h
# Errors only
pup apm traces list --service api-gateway --status error
# Slow traces (>1s)
pup apm traces list --service api-gateway --min-duration 1000ms
# With specific tag
pup apm traces list --query "@http.url:/api/users"
Get Trace Detail
pup apm traces get <trace_id> --json
Key Metrics
| Metric | What It Measures |
|---|---|
trace.http.request.hits | Request count |
trace.http.request.duration | Latency |
trace.http.request.errors | Error count |
trace.http.request.apdex | User satisfaction |
⚠️ Trace Sampling
Not all traces are kept. Understand sampling:
| Mode | What's Kept |
|---|---|
| Head-based | Random % at start |
| Error/Slow | All errors, slow traces |
| Retention | What's indexed (billed) |
# Check retention filters
pup apm retention-filters list
Trace Retention Costs
| Retention | Cost |
|---|---|
| Indexed spans | $$$ per million |
| Ingested spans | $ per million |
Best practice: Only index what you need for search.
Service Level Objectives
Link APM to SLOs:
pup slos create \
--name "API Latency p99 < 200ms" \
--type metric \
--numerator "sum:trace.http.request.hits{service:api,@duration:<200000000}" \
--denominator "sum:trace.http.request.hits{service:api}" \
--target 99.0
Common Queries
| Goal | Query |
|---|---|
| Slowest endpoints | avg:trace.http.request.duration{*} by {resource_name} |
| Error rate | sum:trace.http.request.errors{*} / sum:trace.http.request.hits{*} |
| Throughput | sum:trace.http.request.hits{*}.as_rate() |
Troubleshooting
| Problem | Fix |
|---|---|
| No traces | Check ddtrace installed, DD_TRACE_ENABLED=true |
| Missing service | Verify DD_SERVICE env var |
| Traces not linked | Check trace headers propagated |
| High cardinality | Don't tag with user_id/request_id |