Skip to content

[ERROR] Zero state root hash! #67

@leon-DK

Description

@leon-DK

This error occurs on every node within hours of fully being synced:

2025-10-12|01:37:05.592 [INFO ] Imported new chain segment number=9,884,051 hash=329956..9e7d4e blocks=1 txs=0 mgas=0.000 elapsed=13.881ms mgasps=0.000 snapdiffs=1.96MiB triediffs=656.72KiB triedirty=50.94KiB
2025-10-12|01:37:05.606 [INFO ] Imported new chain segment number=9,884,052 hash=d53f3b..738284 blocks=1 txs=0 mgas=0.000 elapsed=13.868ms mgasps=0.000 snapdiffs=1.96MiB triediffs=535.92KiB triedirty=50.94KiB
2025-10-12|01:37:05.620 [INFO ] Imported new chain segment number=9,884,053 hash=31a027..6f3ce7 blocks=1 txs=0 mgas=0.000 elapsed=14.049ms mgasps=0.000 snapdiffs=1.96MiB triediffs=535.92KiB triedirty=50.94KiB
2025-10-12|01:37:05.620 [INFO ] End DAG REORGANIZE: Old Len= 123;New Len= 131 module=CHAIN
2025-10-12|01:37:05.660 [ERROR] Failed to process block:hash=0x2da50a24fb415b65d2e4bcc9de6100a0ce8051c53d3a1a019ccc29dede4433f5 err=already have block 0x2da50a24fb415b65d2e4bcc9de6100a0ce8051c53d3a1a019ccc29dede4433f5 module=SYNC processID=3636
2025-10-12|01:37:05.660 [ERROR] Failed to process block:hash=0xd721fe69bbd781dbf335f774bef53b6b3083145488b90dc71c24382a0b7564d0 err=already have block 0xd721fe69bbd781dbf335f774bef53b6b3083145488b90dc71c24382a0b7564d0 module=SYNC processID=3636
2025-10-12|01:37:05.663 [ERROR] Failed to process block:hash=0x9664afe61acd5c70d6f2ee29113f31ed91aaae80e28d24f5beea6d73189a2605 err=already have block 0x9664afe61acd5c70d6f2ee29113f31ed91aaae80e28d24f5beea6d73189a2605 module=SYNC processID=3636
2025-10-12|01:37:05.664 [INFO ] The sync of graph state has ended module=SYNC spend=1s processID=3636
2025-10-12|01:37:05.664 [INFO ] You're up to date now. module=SYNC
2025-10-12|01:37:05.664 [INFO ] update BdagPool snapshot module=BDAG size=0
2025-10-12|01:37:06.009 [INFO ] Start DAG REORGANIZE: Block 0x43b15b8d67f2e0e50fbd396c9b71bdb3a6dbf6b32403f27e737a811a671dcf82 is causing a reorganize. module=CHAIN
2025-10-12|01:37:06.009 [INFO ] Try to rewind module=BDAG cur.number=9884053 cur.hash=0x31a0270d4fd01a5ca1dccb652bcbdfececab46f468a15fd3dd20969ebd6f3ce7 target.evm.root=34ebd6..9f325a target.evm.number=9883922 target.evm.hash=a0ddce..df217c
2025-10-12|01:37:06.009 [WARN ] Rewinding blockchain to block target=9,883,922
2025-10-12|01:37:06.009 [ERROR] Zero state root hash!
2025-10-12|01:37:06.009 [INFO ] Rewound to block with state number=9,884,052 hash=d53f3b..738284

...

2025-10-12|01:37:06.078 [INFO ] Rewound to block with state number=9,883,926 hash=055118..e42f82
2025-10-12|01:37:06.079 [ERROR] Zero state root hash!
2025-10-12|01:37:06.079 [INFO ] Rewound to block with state number=9,883,925 hash=db2ada..18dbd1
2025-10-12|01:37:06.079 [ERROR] Zero state root hash!
2025-10-12|01:37:06.079 [CRIT ] Failed to rollback state err="state is unrecoverable"
2025-10-12|01:37:06.162 [ERROR] watchExecuted: subscription error err="websocket: close 1006 (abnormal closure): unexpected EOF"
2025-10-12|01:37:06.163 [INFO ] watchExecuted: reconnecting in 2s
2025-10-12|01:37:06.240 [WARN ] liveness probe failed url=http://127.0.0.1:6061/healthz err="Get "http://127.0.0.1:6061/healthz\": dial tcp 127.0.0.1:6061: connect: connection refused"
2025-10-12|01:37:08.163 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"
2025-10-12|01:37:10.164 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"
2025-10-12|01:37:12.165 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"

root@5c362fad6687:/bdag# bdag --metrics --testnet
2025-10-12|06:40:41.673 [INFO ] System info BDAG Version=1.0.0 Go version=go1.22.12
2025-10-12|06:40:41.673 [INFO ] System info Home dir=.
2025-10-12|06:40:41.673 [INFO ] Using pebble as the backing database module=RAWDB
2025-10-12|06:40:41.752 [INFO ] Opened ancient database module=RAWDB database=data/testnet/BdagChain/ancient/chain readonly=false
2025-10-12|06:40:41.753 [INFO ] transaction index is enabled module=INDEX
2025-10-12|06:40:41.753 [INFO ] anticone size:43 module=DAG
2025-10-12|06:40:41.753 [INFO ] bdag chain module=BDAG version=bdagvm-v0.0.2
2025-10-12|06:40:41.762 [INFO ] New local node record seq=1,760,251,241,761 id=bef4527c57493bd5 ip=127.0.0.1 udp=8538 tcp=0
2025-10-12|06:40:41.765 [INFO ] Prepare bdageth on NetWork(1043)...
2025-10-12|06:40:41.765 [INFO ] Enabling metrics collection
2025-10-12|06:40:41.767 [INFO ] Maximum peer count ETH=0 total=0
2025-10-12|06:40:41.768 [INFO ] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
2025-10-12|06:40:41.770 [INFO ] Set global gas cap cap=50,000,000
2025-10-12|06:40:41.770 [INFO ] Initializing the KZG library backend=gokzg
2025-10-12|06:40:41.782 [INFO ] Allocated trie memory caches clean=154.00MiB dirty=256.00MiB
2025-10-12|06:40:41.782 [INFO ] Using pebble as the backing database
2025-10-12|06:40:41.782 [INFO ] Allocated cache and file handles database=/bdag/data/testnet/bdageth/chaindata cache=512.00MiB handles=536,870,908
2025-10-12|06:40:42.474 [INFO ] Opened ancient database database=/bdag/data/testnet/bdageth/chaindata/ancient/chain readonly=false
2025-10-12|06:40:42.475 [INFO ] State scheme set to already existing scheme=path
2025-10-12|06:40:42.475 [INFO ] Initialising Ethereum protocol network=1043 dbversion=8
2025-10-12|06:40:42.475 [INFO ] Failed to load journal, discard it err="journal not found"
2025-10-12|06:40:42.486 [INFO ] Opened ancient database database=/bdag/data/testnet/bdageth/chaindata/ancient/state readonly=false
2025-10-12|06:40:42.528 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=storage.index items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.531 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=account.data items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.535 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=storage.data items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.537 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=history.meta items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.541 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=account.index items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.545 [WARN ] Truncated extra state histories number=65897
2025-10-12|06:40:42.545 [INFO ]
2025-10-12|06:40:42.545 [INFO ] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2025-10-12|06:40:42.545 [INFO ] Chain ID: 1043 (bdag-test)
2025-10-12|06:40:42.545 [INFO ] Consensus: Ethash (proof-of-work)
2025-10-12|06:40:42.545 [INFO ]
2025-10-12|06:40:42.545 [INFO ] Pre-Merge hard forks (block based):
2025-10-12|06:40:42.545 [INFO ] - Homestead: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
2025-10-12|06:40:42.545 [INFO ] - DAO Fork: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
2025-10-12|06:40:42.545 [INFO ] - Tangerine Whistle (EIP 150): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
2025-10-12|06:40:42.545 [INFO ] - Spurious Dragon/1 (EIP 155): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2025-10-12|06:40:42.545 [INFO ] - Spurious Dragon/2 (EIP 158): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2025-10-12|06:40:42.545 [INFO ] - Byzantium: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
2025-10-12|06:40:42.545 [INFO ] - Constantinople: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
2025-10-12|06:40:42.545 [INFO ] - Petersburg: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
2025-10-12|06:40:42.545 [INFO ] - Istanbul: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
2025-10-12|06:40:42.546 [INFO ] - Muir Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
2025-10-12|06:40:42.546 [INFO ] - Berlin: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
2025-10-12|06:40:42.546 [INFO ] - London: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] The Merge is not yet available for this network!
2025-10-12|06:40:42.546 [INFO ] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] Post-Merge hard forks (timestamp based):
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] Loaded most recent local block number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m22s
2025-10-12|06:40:42.546 [WARN ] Head state missing, repairing number=9,883,925 hash=db2ada..18dbd1 snaproot=40388e..4191e8
2025-10-12|06:40:43.034 [INFO ] Rewound to block with state number=9,818,028 hash=b8682b..10bcc3
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local header number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m23s
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local block number=9,818,028 hash=b8682b..10bcc3 td=9,818,028 age=7h45m14s
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local snap block number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m23s
2025-10-12|06:40:43.041 [WARN ] Enabling snapshot recovery chainhead=9,818,028 diskbase=9,831,120
2025-10-12|06:40:43.042 [WARN ] Loaded snapshot journal diskroot=40388e..4191e8 diffs=unmatched
2025-10-12|06:40:43.042 [WARN ] Snapshot is not continuous with chain snaproot=40388e..4191e8 chainroot=ae45bf..8c289e
2025-10-12|06:40:43.042 [INFO ] Initialized transaction indexer range="last 2350000 blocks"
2025-10-12|06:40:43.042 [INFO ] Setting new local account address=0x250d49eEe9ed2ec5Fdc59d9B0f4Eb8202B03f212
2025-10-12|06:40:43.042 [INFO ] Loaded local transaction journal transactions=1 dropped=0
2025-10-12|06:40:43.042 [INFO ] Regenerated local transaction journal transactions=1 accounts=1
2025-10-12|06:40:43.053 [WARN ] Switch sync mode from snap sync to full sync reason="snap sync complete"
2025-10-12|06:40:43.054 [INFO ] Gasprice oracle is ignoring threshold set threshold=2
2025-10-12|06:40:43.055 [WARN ] Old unclean shutdowns found count=16
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T00:27:48+0000 age=6d6h12m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T08:45:04+0000 age=5d21h55m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T18:12:01+0000 age=5d12h28m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-07T04:49:21+0000 age=5d1h51m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T00:21:18+0000 age=4d6h19m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T02:57:48+0000 age=4d3h42m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T06:41:12+0000 age=3d23h59m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-09T03:37:23+0000 age=3d3h3m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-10T04:17:17+0000 age=2d2h23m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-10T16:42:24+0000 age=1d13h58m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-12T01:34:18+0000 age=5h6m25s
2025-10-12|06:40:43.056 [INFO ] bdag pool init... module=BDAG
2025-10-12|06:40:43.056 [INFO ] Skipping genesis transaction re-indexing for empty block in EVM model module=CDB
2025-10-12|06:40:43.056 [INFO ] Loading dag ... module=CHAIN
2025-10-12|06:40:43.056 [WARN ] get block module=DAG error=empty blockID=10321229
2025-10-12|06:40:43.056 [INFO ] Shutdown complete
2025-10-12|06:40:43.056 [ERROR] The dag data was damaged (Can't find tip:10321229). you can cleanup your block data base by '--cleanup'.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions