xiaohongshu
xiaohongshu
investigate-stuck-messages
Investigate stuck messages in relayer queue. Use when alerts mention "queue length > 0", to diagnose why messages are stuck, or to get message IDs for denylisting.
Full skill instructions
Query the relayer API to investigate stuck messages, their retry counts, and error reasons.
Alert-based triggers:
User request triggers:
Option 1: Grafana Alert URL (recommended)
/investigate-stuck-messages https://abacusworks.grafana.net/alerting/grafana/cdg1ro5hi4vswb/view?tab=instances
Option 2: Manual specification
/investigate-stuck-messages app_context=EZETH/renzo-prod remote=linea
| Parameter | Required | Default | Description |
|---|---|---|---|
alert_url | No | - | Grafana alert URL (extracts app_context/remote from firing instances) |
app_context | No* | - | The app context (e.g., EZETH/renzo-prod, oUSDT/production) |
remote | No* | - | Destination chain name (e.g., linea, ethereum, arbitrum) |
environment | No | mainnet3 | Deployment environment |
*Either alert_url OR both app_context and remote must be provided.
If Grafana alert URL provided:
Extract the alert UID from the URL (e.g., cdg1ro5hi4vswb from .../alerting/grafana/cdg1ro5hi4vswb/view)
Query Prometheus directly for firing instances using mcp__grafana__query_prometheus:
sum by (app_context, remote)(
max_over_time(
hyperlane_submitter_queue_length{
queue_name="prepare_queue",
app_context!~"Unknown|merkly_eth|merkly_erc20|helloworld|velo_message_module",
hyperlane_context!~"rc|vanguard0|vanguard1|vanguard2|vanguard3|vanguard4|vanguard5",
operation_status!~"Retry\\(ApplicationReport\\(.*\\)\\)|FirstPrepareAttempt",
hyperlane_deployment="mainnet3",
}[2m]
)
) > 0
Extract app_context and remote labels from each result.
If manual app_context/remote provided:
Use the provided values directly.
Check if port 9090 is already in use:
lsof -i :9090
If not in use, start port-forward in background:
kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090 -n mainnet3 &
Wait a few seconds for the port-forward to establish.
Look up domain IDs from the registry:
cat node_modules/.pnpm/@hyperlane-xyz+registry@*/node_modules/@hyperlane-xyz/registry/dist/chains/<chain>/metadata.json | jq '.domainId'
Common domain IDs:
For each destination chain, query the relayer API:
curl -s 'http://localhost:9090/list_operations?destination_domain=<DOMAIN_ID>' > /tmp/<chain>.json
The response contains operations with:
id: Message ID (H256)operation.message.sender: Sender addressoperation.message.recipient: Recipient addressoperation.num_retries: Number of retries (higher = more stuck)operation.status: Error status (e.g., {"Retry": "ErrorEstimatingGas"})operation.message.origin: Origin domain IDoperation.message.destination: Destination domain IDoperation.app_context: App context nameLook up the app_context in rust/main/app-contexts/mainnet_config.json:
jq '.metricAppContexts[] | select(.name == "<APP_CONTEXT>")' rust/main/app-contexts/mainnet_config.json
Filter API results to only include messages where:
operation.message.recipient matches one of the recipientAddress values for that destination domainImportant: Addresses are padded to 32 bytes (H256 format).
Calculate log freshness based on retry count:
The relayer uses exponential backoff (see calculate_msg_backoff in rust/main/agents/relayer/src/msg/pending_message.rs):
| Retries | Backoff/retry | Cumulative Time | Freshness Flag |
|---|---|---|---|
| 1-4 | 5s-1min | ~2min | --freshness=1h |
| 5-24 | 3min | ~1h | --freshness=3h |
| 25-39 | 5-26min | ~5h | --freshness=12h |
| 40-49 | 30min-1h | ~12h | --freshness=24h |
| 50-60 | 2-22h | ~35h | --freshness=3d |
| 60+ | 22h+ | 35h+ | --freshness=7d |
For each message ID, query GCP logs with calculated freshness:
gcloud logging read 'resource.type=k8s_container AND resource.labels.namespace_name=mainnet3 AND resource.labels.pod_name:omniscient-relayer AND jsonPayload.span.id:<MESSAGE_ID> AND jsonPayload.fields.error:*' --project=abacus-labs-dev --limit=1 --format='value(jsonPayload.fields.error)' --freshness=<CALCULATED_FRESHNESS>
Extract the human-readable error from the response using sed (macOS compatible):
echo "$raw_error" | sed -n 's/.*execution reverted: \([^"]*\)".*/\1/p' | head -1
Common error patterns:
"execution reverted: Nonce already used" → "Nonce already used""execution reverted: panic: arithmetic underflow" → "Arithmetic underflow"Note: Do not use grep -P as it's not available on macOS.
Output a detailed summary table with full message IDs and both error sources:
## Investigation Results for [APP_CONTEXT]
### Summary
- Total stuck messages: X
- Destinations affected: [list]
- Reprepare reasons: ErrorEstimatingGas (N), CouldNotFetchMetadata (M)
### Messages
| Message ID | Retries | Reprepare Reason | Error | Origin |
|------------|---------|------------------|-----------|--------|
| `0xaa18ebc1c79345e6d24984a0b9a5ab66c968d128d46b2357b641e56e71b8d30c` | 47 | ErrorEstimatingGas | Nonce already used | optimism |
| `0xd6aeef7c092a88aa23ad53227aeb834ae731d059b3ce749db8451e761f3f15ac` | 47 | ErrorEstimatingGas | Nonce already used | arbitrum |
**Important**: Always show the full 66-character message ID (0x + 64 hex chars). Do not truncate.
### Error Analysis
[Explain based on the actual log errors found]
### Next Steps
To denylist these messages, run:
/denylist-stuck-messages <message_ids> app_context=APP_CONTEXT
Column definitions:
operation.status in relayer API (e.g., ErrorEstimatingGas, CouldNotFetchMetadata)At the end of the investigation results, output the full denylist command:
### Next Steps
To denylist, run:
/denylist-stuck-messages 0xaa18ebc1c79345e6d24984a0b9a5ab66c968d128d46b2357b641e56e71b8d30c 0xd6aeef7c092a88aa23ad53227aeb834ae731d059b3ce749db8451e761f3f15ac app_context=APP_CONTEXT
Always use full message IDs, never truncated.
| Status | Meaning | Action |
|---|---|---|
ErrorEstimatingGas | Gas estimation failed (contract revert) | Usually denylist - contract won't accept |
CouldNotFetchMetadata | Can't get ISM metadata | Check validators, may resolve itself |
ApplicationReport(...) | App-specific error | Check the specific error message |
GasPaymentNotFound | No IGP payment | May need manual relay with gas |
kubectl config current-contextkubectl get pods -n mainnet3 | grep relayerkubectl configured with access to mainnet clusterxiaohongshu
technical spec
product ux expert
database patterns
Conduct multi-agent task orchestration and workflow coordination.
Initialize project with Conductor artifacts (product definition,
Expert in web animations, transitions, and motion design using Framer Motion and CSS
Creates Mermaid and ASCII diagrams for flowcharts, architecture, ERDs, state machines, mindmaps, and more. Use when user mentions diagram, flowchart, mermaid, ASCII diagram, text diagram, terminal diagram, visualize, C4, mindmap, architecture diagram, sequence diagram, ERD, or needs visual docume...
PostgreSQL bindings for H3 hexagonal grid system. Use when working with H3 cells in Postgres, including spatial indexing, geometry/geography integration, and raster analysis.
Context-Driven Development skill for projects using Conductor. Use this skill when you detect a `conductor/` directory in the project, when working on tasks defined in a `plan.md` file, or when the user asks about tracks, specs, or plans. Automatically applies TDD workflow, tracks task completion...
Display project status, active tracks, and next actions
Official Stakpak application containerization standard operating procedure, a step-by-step guidline to properly dockerize applications. This is a rule book curated by the Stakpak Team.
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience