correlate-observability-signals
Unify metrics, logs, and traces for cohesive debugging. Implement exemplars for log-to-trace linking, build unified dashboards using RED/USE methods, and enable rapid root cause analysis across observability signals. Use when investigating complex incidents spanning multiple systems, reducing mea...
Full skill instructions
Connect metrics, logs, and traces for unified debugging across the three pillars of observability.
See Extended Examples for complete configuration files and templates.
Add trace IDs to all logs and metrics using OpenTelemetry:
// Go example: Propagate trace context to logs
package main
import (
"context"
"log"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
)
func handleRequest(ctx context.Context, userID string) {
// Extract trace context
span := trace.SpanFromContext(ctx)
traceID := span.SpanContext().TraceID().String()
// Include trace ID in structured logs
log.Printf("trace_id=%s user_id=%s action=process_request", traceID, userID)
// Business logic here
processData(ctx, userID)
}
func processData(ctx context.Context, userID string) {
tracer := otel.Tracer("my-service")
ctx, span := tracer.Start(ctx, "processData")
defer span.End()
traceID := span.SpanContext().TraceID().String()
log.Printf("trace_id=%s user_id=%s action=process_data", traceID, userID)
// More work
}
Python example:
# Python: Flask with OpenTelemetry
from flask import Flask, request
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
import logging
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
logging.basicConfig(
format='%(asctime)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s',
level=logging.INFO
)
@app.route('/api/users/<user_id>')
def get_user(user_id):
span = trace.get_current_span()
trace_id = format(span.get_span_context().trace_id, '032x')
logging.info(f"Fetching user {user_id}", extra={
'otelTraceID': trace_id,
'otelSpanID': format(span.get_span_context().span_id, '016x')
})
# Business logic
return {"user_id": user_id}
Expected: All logs include trace_id field, enabling log-to-trace correlation.
On failure: If trace IDs missing, check OpenTelemetry SDK initialization and context propagation.
Exemplars link metrics to traces:
# prometheus.yml
global:
scrape_interval: 15s
# Enable exemplar storage
exemplars:
max_exemplars: 100000 # Per TSDB block
scrape_configs:
- job_name: 'api-service'
static_configs:
- targets: ['api-service:8080']
# Scrape exemplars
metric_relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds.*'
action: keep
Instrument application to emit exemplars:
// Go: Emit exemplars with Prometheus histogram
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"go.opentelemetry.io/otel/trace"
)
var httpDuration = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request duration",
Buckets: prometheus.DefBuckets,
},
[]string{"method", "endpoint", "status"},
)
func recordRequest(ctx context.Context, method, endpoint, status string, duration float64) {
// Get trace ID for exemplar
span := trace.SpanFromContext(ctx)
traceID := span.SpanContext().TraceID().String()
// Record metric with exemplar
observer := httpDuration.WithLabelValues(method, endpoint, status)
observer.(prometheus.ExemplarObserver).ObserveWithExemplar(
duration,
prometheus.Labels{"trace_id": traceID},
)
}
Query exemplars in Prometheus:
# Histogram with exemplars
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
In Grafana, exemplars appear as dots on histogram graphs that link to traces.
Expected: Grafana shows exemplars in metric graphs, clicking opens corresponding trace.
On failure: Verify Prometheus version ≥2.26 (exemplar support), check Grafana data source config enables exemplars.
RED Method: Rate, Errors, Duration (for services)
{
"dashboard": {
"title": "API Service - RED Dashboard",
"panels": [
{
"title": "Request Rate (req/s)",
"type": "graph",
"targets": [
{
"expr": "sum(rate(http_requests_total{job=\"api-service\"}[5m])) by (endpoint)",
"legendFormat": "{{ endpoint }}"
}
],
"exemplars": true
},
{
"title": "Error Rate (%)",
"type": "graph",
"targets": [
{
"expr": "sum(rate(http_requests_total{job=\"api-service\", status=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"api-service\"}[5m])) * 100",
"legendFormat": "Error %"
}
],
"exemplars": true
},
{
"title": "Request Duration (p50, p95, p99)",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{job=\"api-service\"}[5m]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"api-service\"}[5m]))",
"legendFormat": "p95"
},
{
"expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job=\"api-service\"}[5m]))",
"legendFormat": "p99"
}
],
"exemplars": true
},
{
"title": "Correlated Logs",
"type": "logs",
"datasource": "Loki",
"targets": [
{
"expr": "{job=\"api-service\"} |= \"error\""
}
],
"options": {
"showTime": true,
"enableLogDetails": true
}
}
]
}
}
Expected: Single dashboard showing rate, errors, duration + correlated logs.
On failure: If panels show "No Data", verify metric names match your instrumentation.
USE Method: Utilization, Saturation, Errors (for resources like CPU, memory, disk)
{
"dashboard": {
"title": "Node Resources - USE Dashboard",
"panels": [
{
"title": "CPU Utilization (%)",
"type": "graph",
"targets": [
{
"expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
"legendFormat": "CPU Usage %"
}
]
},
{
"title": "CPU Saturation (Load Average)",
"type": "graph",
"targets": [
{
"expr": "node_load1",
"legendFormat": "1min load"
},
{
"expr": "node_load5",
"legendFormat": "5min load"
},
{
"expr": "count(node_cpu_seconds_total{mode=\"idle\"})",
"legendFormat": "CPU cores (threshold)"
}
]
},
{
"title": "Memory Utilization (%)",
"type": "graph",
"targets": [
{
"expr": "(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100",
"legendFormat": "Memory Usage %"
}
]
},
{
"title": "Memory Saturation (Page Faults)",
"type": "graph",
"targets": [
{
"expr": "rate(node_vmstat_pgmajfault[5m])",
"legendFormat": "Major page faults/s"
}
]
},
{
"title": "Disk Utilization (%)",
"type": "graph",
"targets": [
{
"expr": "(node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes * 100",
"legendFormat": "{{ device }}"
}
]
},
{
"title": "Disk Saturation (IO Wait %)",
"type": "graph",
"targets": [
{
"expr": "rate(node_cpu_seconds_total{mode=\"iowait\"}[5m]) * 100",
"legendFormat": "IO Wait %"
}
]
}
]
}
}
Expected: Dashboard showing resource health across all USE dimensions.
On failure: Ensure node_exporter is running and scraping system metrics.
Configure Loki to extract trace IDs:
# loki-config.yml
schema_config:
configs:
- from: 2024-01-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
# Derived fields for trace linking
query_config:
derived_fields:
- name: TraceID
source: trace_id
url: 'https://tempo.company.com/trace/${__value.raw}'
urlDisplayLabel: 'View Trace'
In Grafana, configure Loki data source:
{
"name": "Loki",
"type": "loki",
"url": "http://loki:3100",
"jsonData": {
"derivedFields": [
{
"datasourceUid": "tempo-uid",
"matcherRegex": "trace_id=(\\w+)",
"name": "TraceID",
"url": "$${__value.raw}"
}
]
}
}
Expected: Clicking trace ID in Loki logs opens corresponding trace in Tempo.
On failure: Verify regex matches your log format, check Tempo data source UID.
Build a dashboard that brings all signals together:
{
"dashboard": {
"title": "Incident Investigation",
"templating": {
"list": [
{
# ... (see EXAMPLES.md for complete configuration)
Workflow during incident:
Expected: Single pane of glass for debugging, jumping between metrics/logs/traces.
On failure: If links don't work, check data source configurations and trace ID propagation.
setup-prometheus-monitoring - metrics foundation for correlationconfigure-log-aggregation - logs foundation for correlationinstrument-distributed-tracing - traces foundation for correlationbuild-grafana-dashboards - unified visualization layer
Perform capacity planning using historical metrics and growth models. Use predict_linear for forecasting, identify resource constraints, calculate headroom, and recommend scaling actions before saturation. Use before seasonal traffic spikes or product launches, during quarterly capacity reviews, ...
Evolve an existing agent definition by refining its persona in-place or creating an advanced variant. Covers assessing the current agent against best practices, gathering evolution requirements, choosing scope (refinement vs. variant), applying changes to skills, tools, capabilities, and limitati...
Version machine learning datasets using DVC (Data Version Control) with remote storage backends, build reproducible data pipelines with dependency tracking, integrate with Git workflows, and ensure data lineage for model reproducibility. Use when versioning large datasets that do not fit in Git, ...
Compose 2D graphics programmatically using SVG generation, diagram layout algorithms, image compositing, and batch processing workflows. Use when generating diagrams, flowcharts, or infographics programmatically, creating reproducible scientific figures, automating production of badges or visual ...
Implement secure secrets management in Kubernetes using SealedSecrets for GitOps, External Secrets Operator for cloud secret managers, and rotation strategies. Handle TLS certificates, API keys, and credentials with encryption at rest and RBAC controls. Use when storing sensitive configuration fo...
Appraise gemstone value using the four Cs (color, clarity, cut, carat), origin assessment, treatment detection, and market factor analysis. Advisory educational guidance only — not a certified appraisal. Use when understanding factors that determine a gemstone's value, pre-screening stones before...
AI dynamic reasoning balance — maintaining grounded reasoning under cognitive pressure, smooth chain-of-thought coordination, and weight-shifting cognitive load across subsystems. Use at the beginning of a complex task requiring multiple coordinated reasoning threads, after a sudden context shift...
Build a competitive or casual trading card game deck. Covers archetype selection, mana/energy curve analysis, win condition identification, meta-game positioning, and sideboard construction for Pokemon TCG, Magic: The Gathering, Flesh and Blood, and other TCGs. Use when building a new deck for a ...
Configure Prometheus for time-series metrics collection, including scrape configurations, service discovery, recording rules, and federation patterns for multi-cluster deployments. Use when setting up centralized metrics collection for microservices, implementing time-series monitoring for applic...
Conduct temperament-based health assessment from Hildegard von Bingen's Causae et Curae. Evaluates the four temperaments (sanguine, choleric, melancholic, phlegmatic), elemental correspondences (air, fire, earth, water), and provides dietary and lifestyle recommendations for rebalancing. Use when...
Implement GitOps continuous delivery using Argo CD or Flux with app-of-apps pattern, automated sync policies, drift detection, and multi-environment promotion. Manage Kubernetes deployments declaratively from Git with automated reconciliation. Use when implementing declarative infrastructure mana...
Map the intellectual property landscape for a technology domain or product area. Covers patent cluster analysis, white space identification, competitor IP portfolio assessment, freedom-to-operate preliminary screening, and strategic IP positioning recommendations. Use before starting R&D in a new...
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience