Skip to content
investigate-merkle-root-mismatch logo

Investigate Merkle Root Mismatch

investigate-merkle-root-mismatch

Investigate merkle root mismatch alerts between relayer and validators. Use when alerts mention "merkle root mismatch", "checkpoint root does not match canonical root", or when asked to debug relayer merkle tree issues for a chain. This skill only investigates - use /fix-merkle-root-mismatch to a...

SKILL.md

Full skill instructions

Investigate Merkle Root Mismatch

When to Use

  1. Alert-based triggers:

    • Alert mentions "merkle root mismatch"
    • GCP logs show: "checkpoint root does not match canonical root from merkle proof"
    • The hyperlane_merkle_root_mismatch metric is firing
  2. User request triggers:

    • "Debug merkle tree issues for [chain]"
    • "Investigate the merkle root mismatch on [chain]"
    • "Why is the relayer's merkle tree wrong for [chain]?"

Input Parameters

ParameterRequiredDefaultDescription
originYes-The origin chain with merkle root mismatch (e.g., paradex, ethereum)
domain_idNo-Domain ID of the chain (auto-derived from registry if not provided)
environmentNomainnet3mainnet3 or testnet4

Problem Overview

The relayer maintains a local merkle tree built from message IDs. When a validator signs a checkpoint, the relayer needs to generate a merkle proof. If the relayer's tree has incorrect message IDs, the roots will mismatch and message delivery will fail.

Most likely cause: The relayer indexed incorrect message IDs (possibly due to RPC issues or reorgs), while validators have the correct data.

Prerequisites

  • kubectl access to the relayer pods
  • Grafana MCP server configured

Investigation Workflow

Step 1: Confirm the Alert

Query Grafana to confirm the mismatch metric is firing:

Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_merkle_root_mismatch{origin="[origin]"}
- startTime: now-1h
- queryType: instant

If value is 1, the mismatch is confirmed.

Step 2: Get Latest Tree Insertion Index

Query Grafana for the current tree size:

Use mcp__grafana__query_prometheus with:
- datasourceUid: grafanacloud-prom
- expr: hyperlane_latest_tree_insertion_index{origin="[origin]", hyperlane_deployment="[environment]"}
- startTime: now-1h
- queryType: instant

This gives you the latest leaf index to work backwards from.

Step 3: Get Domain ID

If domain_id was not provided, fetch from the registry:

curl -s "https://raw.githubusercontent.com/​hyperlane-xyz/​hyperlane-registry/​main/​chains/[origin]/​metadata.yaml" | grep domainId

Step 4: Establish Port-Forward to Relayer

Check if port 9090 is already in use:

lsof -i :9090

If not in use, start port-forward in background:

kubectl port-forward omniscient-relayer-hyperlane-agent-relayer-0 9090:9090 -n [environment] &

Wait a few seconds for the port-forward to establish, then verify it's working:

curl -s "localhost:9090/​merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=0&leaf_index_end=1"

Step 5: Binary Search for First Mismatch

Compare validator checkpoints (from S3) against relayer merkle proofs. Use binary search to find the FIRST mismatched index.

Validator checkpoint URL pattern:

https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/​checkpoint_[index]_with_id.json

Comparison function:

# Check a specific index
index=[INDEX]
domain_id=[DOMAIN_ID]
origin=[ORIGIN]
environment=[ENVIRONMENT]

validator_root=$(curl -s "https://hyperlane-${environment}-${origin}-validator-0.s3.us-east-1.amazonaws.com/​checkpoint_${index}_with_id.json" | jq -r '.value.checkpoint.root')
relayer_root=$(curl -s "localhost:9090/​merkle_proofs?domain_id=${domain_id}&leaf_index=${index}&root_index=${index}" | jq -r '.root')

echo "Index $index:"
echo "  Validator: $validator_root"
echo "  Relayer:   0x$relayer_root"
if [ "$validator_root" = "0x$relayer_root" ]; then echo "  ✓ Match"; else echo "  ❌ MISMATCH"; fi

Binary search strategy:

  1. Start at the latest index - if mismatch, go to 50% of that index
  2. If match at 50%, mismatch is between 50%-100% - try 75%
  3. If mismatch at 50%, mismatch is between 0%-50% - try 25%
  4. Continue narrowing until you find the exact first mismatch index (where index N-1 matches but index N mismatches)

Step 6: Identify Mismatched Message IDs

Once you found the first mismatch index, compare message IDs:

Get validator message IDs:

for i in $(seq [first_mismatch] [first_mismatch + 10]); do
  msg=$(curl -s "https://hyperlane-[environment]-[origin]-validator-0.s3.us-east-1.amazonaws.com/​checkpoint_${i}_with_id.json" | jq -r '.value.message_id')
  echo "$i: $msg"
done

Get relayer message IDs:

curl -s "localhost:9090/​merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=[first_mismatch]&leaf_index_end=[first_mismatch + 10]" | jq -r '.merkle_tree_insertions[] | "\(.leaf_index): \(.message_id)"'

Step 7: Get Block Timestamp for Context

Get the block number and timestamp of the first mismatch to understand when the issue started:

# Get block number from relayer
curl -s "localhost:9090/​merkle_tree_insertions?domain_id=[domain_id]&leaf_index_start=[first_mismatch]&leaf_index_end=[first_mismatch]" | jq '.merkle_tree_insertions[0].insertion_block_number'

For EVM chains, get timestamp:

cast block [block_number] --rpc-url [rpc_url] -j | jq '.timestamp'

For Starknet chains:

curl -s --request POST --url '[rpc_url]' --header 'Content-Type: application/​json' --data '{"jsonrpc":"2.0","method":"starknet_getBlockWithTxHashes","params":[{"block_number":[block_number]}],"id":1}' | jq '.result.timestamp'

Convert Unix timestamp to human-readable:

date -r [timestamp] -u '+%Y-%m-%d %H:%M:%S UTC'

Step 8: Report Findings

Present the investigation results with:

  1. Summary table:
ParameterValue
Chain[origin]
Domain ID[domain_id]
Environment[environment]
First Mismatch Index[index]
Latest Index[latest]
Total Entries to Fix[latest - first_mismatch + 1]
Mismatch Started At[block_number] ([timestamp])
  1. Sample of mismatched entries:
Leaf IndexRelayer Message IDValidator Message IDBlock Number
[idx]0x...0x...[block]
  1. Inform the user that to fix this issue, they should run /​fix-merkle-root-mismatch.

  2. Note about fixing: ALL entries from the first mismatch to the latest must be fixed because the merkle tree is cumulative - each root depends on all previous leaves.

API Reference

Relayer Endpoints

EndpointMethodParametersDescription
/​merkle_tree_insertionsGETdomain_id, leaf_index_start, leaf_index_endList merkle tree insertions
/​merkle_proofsGETdomain_id, leaf_index, root_indexGet merkle proof for a leaf

Validator S3 Checkpoint

https://hyperlane-[environment]-[chain]-validator-0.s3.us-east-1.amazonaws.com/​checkpoint_[index]_with_id.json

Response structure:

{
  "value": {
    "checkpoint": {
      "merkle_tree_hook_address": "0x...",
      "mailbox_domain": 514051890,
      "root": "0x...",
      "index": 37352
    },
    "message_id": "0x..."
  },
  "signature": { ... }
}

Common Issues

  1. Port-forward disconnects: Re-run the kubectl port-forward command
  2. Validator S3 returns 404: Checkpoint may not exist yet at that index
  3. Binary search takes too long: Use larger jumps initially (e.g., 10000 indices)
  4. Shell script errors: Use manual curl commands instead of the bash scripts in rust/​scripts/

Next Steps

After investigation, use /​fix-merkle-root-mismatch to apply the fixes.

Runbook Reference

Full runbook: https://www.notion.so/hyperlanexyz/Merkle-Root-Mismatch-26a6d35200d680a2857dcd0b228d4ab7

More DevOps & CI/CD skills

Project scaffolding, deployment configuration, and CI/CD setup for Google ADK agents.

6K 472.6K
View

Set up tracing, logging, and monitoring for deployed ADK agents across Cloud Trace, BigQuery, and third-party platforms.

6K 472.6K
View

Enterprise Azure infrastructure architect generating Bicep or Terraform from workload descriptions.

1.5K 454.3K
View
azure-kubernetes logo
DevOps & CI/CD

azure-kubernetes

Plan and configure production-ready Azure Kubernetes Service clusters with Day-0 and Day-1 best practices.

1.5K 447.1K
View

Raw mechanical interfaces fusing Swiss typographic print with military terminal aesthetics. Rigid grids, extreme type scale contrast, utilitarian color, analog degradation effects. For data-heavy dashboards, portfolios, or editorial sites that need to feel like declassified blueprints.

92.7K 354.9K
View
just-scrape logo
DevOps & CI/CD

just-scrape

Web search, scraping, extraction, crawling, and monitoring via ScrapeGraph AI CLI.

69 245K
View

Skill for working with Firebase Hosting (Classic). Use this when you want to deploy static web apps, Single Page Apps (SPAs), or simple microservices. Do NOT use for Firebase App Hosting.

462 159.7K
View

Deploy and manage web apps with Firebase App Hosting. Use this skill when deploying Next.js/Angular apps with backends.

462 159.1K
View
deploy-to-vercel logo
DevOps & CI/CD

deploy-to-vercel

Deploy applications and websites to Vercel. Use when the user requests deployment actions like "deploy my app", "deploy and give me the link", "push this live", or "create a preview deployment".

31.9K 146.6K
View
programmatic-seo logo
DevOps & CI/CD

programmatic-seo

Build SEO-optimized pages at scale using templates, data, and proven playbook patterns.

53.3K 140.1K
View

Design and build isolated, reusable Convex backend components with clear boundaries and app-facing wrappers.

63 123.5K
View

Deploy and manage projects on Vercel using token-based authentication. Use when working with Vercel CLI using access tokens rather than interactive login — e.g. "deploy to vercel", "set up vercel", "add environment variables to vercel".

31.9K 116.1K
View

Coding & apps AI tools

Opus Clip logo
Coding & apps

Opus Clip

Opus.ai: Revolutionize Your Web Experience

Free
View
Imagica logo
Coding & apps

Imagica

Build a no-code AI app in minutes.

Freemium
View
E
Coding & apps

Emergent.sh

An IDE for code migration from legacy to modern frameworks through coding agents.

Freemium
View
Wonder Dynamics logo
Coding & apps

Wonder Dynamics

Automate CGI animation in live-action scenes

Paid
View
M
Coding & apps

Mixo

Launch a website in seconds with AI.

Paid
View
AI Code Convert logo
Coding & apps

AI Code Convert

Streamline Your Coding Experience with AI Code Helper

Free
View