google-agents-cli-scaffold
Project scaffolding, deployment configuration, and CI/CD setup for Google ADK agents.
investigate-merkle-root-mismatch
Investigate merkle root mismatch alerts between relayer and validators. Use when alerts mention "merkle root mismatch", "checkpoint root does not match canonical root", or when asked to debug relayer merkle tree issues for a chain. This skill only investigates - use /fix-merkle-root-mismatch to a...
Full skill instructions
Alert-based triggers:
hyperlane_merkle_root_mismatch metric is firingUser request triggers:
| Parameter | Required | Default | Description |
|---|---|---|---|
origin | Yes | - | The origin chain with merkle root mismatch (e.g., paradex, ethereum) |
domain_id | No | - | Domain ID of the chain (auto-derived from registry if not provided) |
environment | No | mainnet3 | mainnet3 or testnet4 |
The relayer maintains a local merkle tree built from message IDs. When a validator signs a checkpoint, the relayer needs to generate a merkle proof. If the relayer's tree has incorrect message IDs, the roots will mismatch and message delivery will fail.
Most likely cause: The relayer indexed incorrect message IDs (possibly due to RPC issues or reorgs), while validators have the correct data.
kubectl access to the relayer podsQuery Grafana to confirm the mismatch metric is firing:
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_merkle_root_mismatch{origin="[origin]"}
- startTime: now-1h
- queryType: instant
If value is 1, the mismatch is confirmed.
Query Grafana for the current tree size:
Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_latest_tree_insertion_index{origin="[origin]", hyperlane_deployment="[environment]"}
- startTime: now-1h
- queryType: instant
This gives you the latest leaf index to work backwards from.
If domain_id was not provided, fetch from the registry:
curl -s "https://raw.githubusercontent.com/hyperlane-xyz/hyperlane-registry/main/chains/[origin]/metadata.yaml" | grep domainId
Check if port 9090 is already in use:
lsof -i :9090
If not in use, start port-forward in background:
kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090:9090 -n [environment] &
Wait a few seconds for the port-forward to establish, then verify it's working:
curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=0&leaf_index_end=1"
Compare validator checkpoints (from S3) against relayer merkle proofs. Use binary search to find the FIRST mismatched index.
Validator checkpoint URL pattern:
https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_[index]_with_id.json
Comparison function:
# Check a specific index
index=[INDEX]
domain_id=[DOMAIN_ID]
origin=[ORIGIN]
environment=[ENVIRONMENT]
validator_root=$(curl -s "https://hyperlane-${environment}-${origin}-validator-0.s3.us-east-1.amazonaws.com/checkpoint_${index}_with_id.json" | jq -r '.value.checkpoint.root')
relayer_root=$(curl -s "localhost:9090/merkle_proofs?domain_id=${domain_id}&leaf_index=${index}&root_index=${index}" | jq -r '.root')
echo "Index $index:"
echo " Validator: $validator_root"
echo " Relayer: 0x$relayer_root"
if [ "$validator_root" = "0x$relayer_root" ]; then echo " ✓ Match"; else echo " ❌ MISMATCH"; fi
Binary search strategy:
Once you found the first mismatch index, compare message IDs:
Get validator message IDs:
for i in $(seq [first_mismatch] [first_mismatch + 10]); do
msg=$(curl -s "https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_${i}_with_id.json" | jq -r '.value.message_id')
echo "$i: $msg"
done
Get relayer message IDs:
curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=[first_mismatch]&leaf_index_end=[first_mismatch + 10]" | jq -r '.merkle_tree_insertions[] | "\(.leaf_index): \(.message_id)"'
Get the block number and timestamp of the first mismatch to understand when the issue started:
# Get block number from relayer
curl -s "localhost:9090/merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=[first_mismatch]&leaf_index_end=[first_mismatch]" | jq '.merkle_tree_insertions[0].insertion_block_number'
For EVM chains, get timestamp:
cast block [block_number] --rpc-url [rpc_url] -j | jq '.timestamp'
For Starknet chains:
curl -s --request POST --url '[rpc_url]' --header 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"starknet_getBlockWithTxHashes","params":[{"block_number":[block_number]}],"id":1}' | jq '.result.timestamp'
Convert Unix timestamp to human-readable:
date -r [timestamp] -u '+%Y-%m-%d %H:%M:%S UTC'
Present the investigation results with:
| Parameter | Value |
|---|---|
| Chain | [origin] |
| Domain ID | [domain_id] |
| Environment | [environment] |
| First Mismatch Index | [index] |
| Latest Index | [latest] |
| Total Entries to Fix | [latest - first_mismatch + 1] |
| Mismatch Started At | [block_number] ([timestamp]) |
| Leaf Index | Relayer Message ID | Validator Message ID | Block Number |
|---|---|---|---|
| [idx] | 0x... | 0x... | [block] |
Inform the user that to fix this issue, they should run /fix-merkle-root-mismatch.
Note about fixing: ALL entries from the first mismatch to the latest must be fixed because the merkle tree is cumulative - each root depends on all previous leaves.
| Endpoint | Method | Parameters | Description |
|---|---|---|---|
/merkle_tree_insertions | GET | domain_id, leaf_index_start, leaf_index_end | List merkle tree insertions |
/merkle_proofs | GET | domain_id, leaf_index, root_index | Get merkle proof for a leaf |
https://hyperlane-[environment]-[chain]-validator-0.s3.us-east-1.amazonaws.com/checkpoint_[index]_with_id.json
Response structure:
{
"value": {
"checkpoint": {
"merkle_tree_hook_address": "0x...",
"mailbox_domain": 514051890,
"root": "0x...",
"index": 37352
},
"message_id": "0x..."
},
"signature": { ... }
}
rust/scripts/After investigation, use /fix-merkle-root-mismatch to apply the fixes.
Full runbook: https://www.notion.so/hyperlanexyz/Merkle-Root-Mismatch-26a6d35200d680a2857dcd0b228d4ab7
Project scaffolding, deployment configuration, and CI/CD setup for Google ADK agents.
Set up tracing, logging, and monitoring for deployed ADK agents across Cloud Trace, BigQuery, and third-party platforms.
Enterprise Azure infrastructure architect generating Bicep or Terraform from workload descriptions.
Plan and configure production-ready Azure Kubernetes Service clusters with Day-0 and Day-1 best practices.
Raw mechanical interfaces fusing Swiss typographic print with military terminal aesthetics. Rigid grids, extreme type scale contrast, utilitarian color, analog degradation effects. For data-heavy dashboards, portfolios, or editorial sites that need to feel like declassified blueprints.
Web search, scraping, extraction, crawling, and monitoring via ScrapeGraph AI CLI.
Skill for working with Firebase Hosting (Classic). Use this when you want to deploy static web apps, Single Page Apps (SPAs), or simple microservices. Do NOT use for Firebase App Hosting.
Deploy and manage web apps with Firebase App Hosting. Use this skill when deploying Next.js/Angular apps with backends.
Deploy applications and websites to Vercel. Use when the user requests deployment actions like "deploy my app", "deploy and give me the link", "push this live", or "create a preview deployment".
Build SEO-optimized pages at scale using templates, data, and proven playbook patterns.
Design and build isolated, reusable Convex backend components with clear boundaries and app-facing wrappers.
Deploy and manage projects on Vercel using token-based authentication. Use when working with Vercel CLI using access tokens rather than interactive login — e.g. "deploy to vercel", "set up vercel", "add environment variables to vercel".
Opus.ai: Revolutionize Your Web Experience
Build a no-code AI app in minutes.
An IDE for code migration from legacy to modern frameworks through coding agents.
Automate CGI animation in live-action scenes
Branded artistic QR-code concepts
Launch a website in seconds with AI.
Streamline Your Coding Experience with AI Code Helper
Convert any screenshot or design to clean code.