How to Troubleshoot Your Hyperlane Deployment
General Troubleshooting
Logging
Logging levels and formats can be configured for Hyperlane tooling using the following two environment variables:
LOG_LEVEL
: The log level to filter to. Defaults toinfo
. Choices:debug | info | warn | error | off
LOG_FORMAT
: The format of the log output. Defaults topretty
. Choices:pretty | json
The Hyperlane CLI also allows configuring logging via the --log
and --verbosity
flags.
Chain configuration
Within your working directory, you may find a chains/
yaml files organized by chain name. These metadata.yaml
files describe the information needed to use the chain in Hyperlane deployments and apps.
You can define a full configuration for any new chain in this file. The metadata that can be configured is defined in this example configuration. You can also find the chain metadata schema at chainMetadataTypes.ts.
Here's an example configuration for two local anvil chains:
Example: Two Local Anvil Chains
anvil1:
chainId: 31337
domainId: 31337
name: anvil1
protocol: ethereum
rpcUrls:
- http: http://localhost:8545
nativeToken:
name: Ether
symbol: ETH
decimals: 18
anvil2:
chainId: 31338
domainId: 31338
name: anvil2
protocol: ethereum
rpcUrls:
- http: http://localhost:8555
You can also extend a core chain config by providing the fields to be overridden:
sepolia:
blocks:
confirmations: 2
Override RPC URLs
You can override the RPC urls by extending the core chain config.
Example: Define RPC URLs array and adjust retry settings
demochain:
name: demochain
chainId: 123456
domainId: 123456
protocol: ethereum
rpcUrls:
- http: https://rpc-testnet.demochain.gg
sepolia:
rpcUrls:
- http: https://rpc2.sepolia.org
- http: https://some-other-sepolia-rpc.gg
retry:
maxRequests: 10
Override transaction settings
Transaction overrides are any properties to include when forming transaction requests. For example:
gasPrice
: number | stringmaxFeePerGas
: number | stringmaxPriorityFeePerGas
: number | stringnonce
: number | stringtype
: numberccipReadEnabled
: boolean
In the example below we're using a gas price of 7 gwei, hardcoding the nonce, and setting a maximum value for the base and priority fees.
sepolia:
transactionOverrides:
gasPrice: 7000000000 # 7 gwei
maxFeePerGas: 150000000000 # 150 gwei
maxPriorityFeePerGas: 40000000000 # 40 gwei
nonce: 1337
If you are overriding the nonce in the chain configuration, ensure you are updating the value on successful transactions.
Agent Troubleshooting
Validator Troubleshooting
-
Validator has not announced any storage locations
Warning:- Reason: This occurs when the relayer cannot retrieve validator signatures required to process a message.
- Solution:
- Ensure the validators have announced their storage locations. Check validator logs for a message such as:
Validator has announced signature storage location, locations: ["file:///tmp/hyperlane-validator-signatures-local"]
- Verify that each validator has a unique signature storage path (
--checkpointSyncer.path
) to prevent overwriting. - Confirm that the relayer has read access to the storage paths.
- Ensure the validators have announced their storage locations. Check validator logs for a message such as:
-
“No ISM Found for Origin" Error
- Reason: This occurs when the Interchain Security Module (ISM) on your chain does not recognize the origin chain.
- Solution:
- Ensure the ISM configuration includes the Validators for the origin chain.
- If it doesn't, add validators for the origin chain to your ISM.
- Restart the relayer after updating the ISM configuration.
Relayer Troubleshooting
- Relayer Selecting Incorrect ISM
- Reason: This error occurs when the Interchain Security Module (ISM) on your chain does not recognize the origin chain.
- Solution:
- Check your chain's ISM deployment configuration.
- Set the ISM type during deployment to ensure the correct one is used:
{ "ismType": "messageIdMultisig", "validators": ["<validator_address>"] }
- Verify the ISM address in the relayer configuration.
Advanced Compatibility Troubleshooting
eth_getStorageAt()
Compatibility
Some chains do not natively support the eth_getStorageAt()
API. If you're deploying on one of these chains and encounter issues, review the changes made to the Hyperlane codebase in this commit.
To deploy Hyperlane on these chains follow the steps: