Troubleshooting

Detailed guide for troubleshooting node problems.

Knowing common errors and their solutions helps keep your node healthy.

Common error codes

These are the most frequent errors and their solutions:

Consensus errors

If you get a consensus error, act quickly and appropriately:

Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status

Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json

Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines

Network errors

Network errors can prevent your node from participating in consensus:

Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status

Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info

Database errors

Database corruption can require immediate attention:

Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/

Diagnostic commands

These commands help you investigate problems and monitor your node:

# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)

AppHash mismatch errors

If you get an AppHash mismatch, capture the state to compare it with a known-good version:

# For SeiDB (all supported nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid

As with FlatKV, the memIAVL store has two possible locations. Nodes created before the layout change keep the legacy $HOME/.sei/data/committer.db path shown above (it takes precedence when present). New nodes use $HOME/.sei/data/state_commit/memiavl. Point -d at whichever path exists on your node.

On Giga Storage nodes, EVM state lives in a FlatKV store instead of the memIAVL trees. An AppHash comparison there also requires a FlatKV state dump. Use the dump-flatkv command to iterate and dump the physical (key, value) pairs into per-bucket files. The files match the dump-iavl format, so the same diff tooling works on both:

# For FlatKV (Giga Storage nodes hold EVM state in FlatKV):
systemctl stop seid
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump
systemctl restart seid

Nodes created before the storage layout change keep the FlatKV store at the legacy path $HOME/.sei/data/flatkv instead of $HOME/.sei/data/state_commit/flatkv. Pass whichever directory exists on your node to --db-dir.

dump-flatkv reads from a temporary copy of the store, so you can also run it against a live node. For all flags, see Dumping FlatKV state in the technical reference.

For example, to dump only the storage bucket at a specific version:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage

To check only the FlatKV lattice hash against the snapshot metadata, with no dump files:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --lthash-only

If the lattice hash matches, the output shows result: PASS. On a mismatch, the command prints TOTAL FAIL and stops with an error.

Comparing EVM state between memIAVL and FlatKV

When you debug an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends, even when the underlying state is identical. This happens because every FlatKV value embeds a per-key block-height stamp (the height at which the key was last written or migrated). On a freshly migrated node, this stamp differs from the memIAVL leaf versions.

The evm-logical-digest command works around this with a backend-independent digest of the EVM logical state. On both sides, it strips the serialization-version and block-height header. Then it digests only the logical payload: account balance, nonce, and code hash, plus bytecode and storage words. With this digest, you can compare a memIAVL node and a FlatKV node at the same chain height:

# FlatKV digest at a height (WAL-replays to it):
seidb evm-logical-digest --backend flatkv \
    --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000

# memIAVL digest at the same height (0 = current symlink):
seidb evm-logical-digest --backend memiavl \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000

The memIAVL digest above needs a snapshot at the target height. If no snapshot exists at that height, add --memiavl-open-mode changelog. This mode reads the nearest older snapshot and applies the changelog up to the height. The replay mode returns the same rows, but it is about 10 times slower.

# memIAVL digest for a height with no on-disk snapshot:
seidb evm-logical-digest --backend memiavl --memiavl-open-mode changelog \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213205000

On a node in the middle of the migration, EVM state is split between FlatKV and memIAVL. Use --backend composite to digest both together, so you can compare the node with a memIAVL-only node at the same height. Pass the two directories with --flatkv-dir and --memiavl-dir instead of --db-dir:

# Mid-migration node: digest the flatkv + memiavl union at one height:
seidb evm-logical-digest --backend composite --memiavl-open-mode changelog \
    --flatkv-dir $HOME/.sei/data/state_commit/flatkv \
    --memiavl-dir $HOME/.sei/data/state_commit/memiavl --height 213200000

Each run prints per-bucket bucket_digest values and a single FINAL_DIGEST line that covers the account, code, storage, and misc buckets. Compare the FINAL_DIGEST lines from both backends at the same height. If the EVM state is the same, the lines match. The command leaves out FlatKV’s internal migration rows, so memIAVL, mid-migration, and migrated nodes all produce comparable digests.

For targeted debugging, the command can also inspect a single normalized bucket instead of printing the global digest. The seidb tooling section of the technical reference has the full flag reference, including inspect mode, sharding, and --find-hash. These examples list the first 50 account rows with version metadata, and shard the storage bucket under a key prefix by the next 2 bytes:

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket account --list --list-limit 50 --details

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2

Enable SeiDB State Commit (SC). Set sc-enable = true in the [state-commit] section of app.toml. The legacy IAVL backend was fully removed, and SC is now mandatory. If SC is not enabled, the node no longer falls back to IAVL. It panics at startup with this error:

SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated

The seid debug dump-iavl command was also removed with the IAVL backend. To inspect state, use the seidb dump-iavl tool shown above.

When you report a problem, always include the app hash, commit hash, and block height from your logs.

Identifying AppHash errors

In logs, AppHash errors usually look like this:

ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]

Common causes:

• Using an incorrect node version during sync (make sure that you run the latest version)
• Corrupted or incorrectly applied snapshots
• Database inconsistencies from improper shutdowns
• Syncing with outdated or incompatible peers

Resolution steps:

1. Stop the node immediately.

2. Try a node rollback first. See Node rollback.

3. If the rollback fails, restore from a fresh snapshot:

   • Download a recent snapshot from trusted providers (Polkachu, PublicNode)
   • Make sure that you use the correct node version
   • Verify that peer configurations are up to date

4. Restart the node and monitor the logs for continued errors.

Peer connection issues as AppHash red herrings

Important: Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

If you see many peer connection errors like these:

ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF

Do not focus only on fixing peer connections first. Instead:

1. Scan your logs carefully for AppHash errors that may appear intermittently
2. Look for the actual error pattern:
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   
3. Check whether your node is stuck at a specific height despite peer connection attempts

Why this happens:

• AppHash mismatches prevent proper block validation
• The node cannot advance to new blocks because of a state inconsistency
• Peers may reject connections from nodes with corrupted state
• The network appears to be the problem, but the cause is a local state issue

Debugging approach:

1. First, check for AppHash errors in your logs (search for “wrong Block.Header.AppHash”)
2. If you find AppHash errors, treat them as the primary issue
3. Focus on peer connection fixes only if no AppHash errors exist

This approach targets the root cause, not the symptoms, and can save hours of debugging time.

Peer connection and handshake issues

Identifying peer issues:

Look for these error patterns in your logs:

ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p

Common causes:

• Outdated peer configurations with mismatched node IDs
• Network infrastructure changes on the peer side
• A firewall that blocks connections on port 26656
• DNS resolution issues

Resolution steps:

1. Reset peer configuration to the Sei Labs seeds. These are stable published addresses, so a stale or mismatched node ID is the usual cause of this error:

   On seid v6.6.2 and later, seid init writes these for you. Re-running init against an existing home will not overwrite a bootstrap-peers value you have already set, so edit config.toml directly if you need to replace one.

2. Verify network connectivity:

   # Test connection to peer endpoints
   nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656
   
   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   

3. Check the current peer status:

   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   

Sync performance issues

Identifying sync problems:

Monitor these indicators:

# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'

Common solutions:

1. Increase the packet payload size for large block processing:

   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000  # Increase from default 102400
   

2. Optimize the mempool settings in config.toml:

   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   

3. If the node gets stuck at a specific height:

   • Try restarting the node
   • If that does not help, perform a rollback
   • Consider taking a fresh snapshot

Warning signs to watch for:

• The current height does not increase over time
• Increasing lag between the current height and the max peer height
• Repeated timeout errors in the logs
• Mempool size that consistently reaches its limits

Crash and panic debugging

For crashes, panics, or nil pointer exceptions:

• Capture at least 1,000 lines of logs before the crash or 15 minutes of log data, whichever gives more context
• Include the full stack trace, if it is available

Logging configuration

Proper logging configuration is essential for debugging and monitoring:

# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs

Configure log rotation to manage storage:

# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF

Enable core dumps for crash analysis:

# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern

Other common issues and fixes

1. Sync problems

   • Check available disk space (df -h)
   • Make sure that peer connections work (curl http://localhost:26657/net_info)
   • Check that the firewall allows port 26656

2. Performance issues

   • Monitor system resources (htop or iotop)
   • Check disk I/O performance (iostat)
   • Analyze network traffic (iftop)

3. Database issues

   • Run database integrity checks:

     seid debug dump-db | grep -i error
     

     If you find errors, consider restoring from a recent backup.

   • To keep less historical data, lower ss-keep-recent in app.toml.

   • To rebuild the node with a smaller database, reset it. Then resync from a snapshot or with state sync. The reset deletes all chain data, so it is not a pruning tool. Before you reset, back up priv_validator_key.json and priv_validator_state.json, as described in Clean up:

     seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
     

     Alternatively, remove old state snapshots manually to free disk space:

     rm -rf $HOME/.sei/data/snapshots/*
     

Node rollback

To roll back a node from an AppHash mismatch, first stop the node in your preferred way.

Next, roll back the node:

seid rollback

Then, restart the node.

If you see this error when you try to roll back:

failed to initialize database: resource temporarily unavailable

This means that you did not shut down the node properly. In that case, try to shut down or kill the seid process directly. If this does not help, restart your machine.

Then try the rollback steps again.

HashVault app-hash equivocation panic

Under Autobahn, HashVault halts a node that tries to commit a different app hash for a height it already finalized. For background, see HashVault.

Before the node halts, it logs an error like this:

Hashvault detected app hash mismatch; node attempted to change its mind. DO NOT RESTART WITHOUT HUMAN INVESTIGATION. ...
blockHeight=[HEIGHT] existingHex=[...] incomingHex=[...] hashVaultDir=[/path/to/hashvault]

Do not restart the node before you investigate. A mismatch can mean a real equivocation, for example the same validator key on two machines. If you restart the node or remove the guard, your validator can commit a conflicting hash for a finalized height. This can cause slashing.

Find out if the mismatch is a real equivocation. A rollback or restore outside the normal process can also leave the app state out of sync with the stored hashes.

Recovery

Only if you are sure that there is no real equivocation:

1. Stop the node.
2. Delete the HashVault directory shown as hashVaultDir in the log (by default <PersistentStateDir>/hashvault).
3. Restart the node. It starts with an empty hash history.

Deleting the directory removes the protection for the heights it covered.

Disabling HashVault

If the same panic occurs again on new blocks, and you are sure that the stored hashes are wrong, you can turn off HashVault. Set the top-level hash-vault-disabled-unsafe key in config.toml:

# config.toml (top-level, before any [section] header)
hash-vault-disabled-unsafe = true

The node then runs without equivocation protection and logs errors at every start. Set the key back to false as soon as you fix the problem.