-
Notifications
You must be signed in to change notification settings - Fork 4
Description
This error occurs on every node within hours of fully being synced:
2025-10-12|01:37:05.592 [INFO ] Imported new chain segment number=9,884,051 hash=329956..9e7d4e blocks=1 txs=0 mgas=0.000 elapsed=13.881ms mgasps=0.000 snapdiffs=1.96MiB triediffs=656.72KiB triedirty=50.94KiB
2025-10-12|01:37:05.606 [INFO ] Imported new chain segment number=9,884,052 hash=d53f3b..738284 blocks=1 txs=0 mgas=0.000 elapsed=13.868ms mgasps=0.000 snapdiffs=1.96MiB triediffs=535.92KiB triedirty=50.94KiB
2025-10-12|01:37:05.620 [INFO ] Imported new chain segment number=9,884,053 hash=31a027..6f3ce7 blocks=1 txs=0 mgas=0.000 elapsed=14.049ms mgasps=0.000 snapdiffs=1.96MiB triediffs=535.92KiB triedirty=50.94KiB
2025-10-12|01:37:05.620 [INFO ] End DAG REORGANIZE: Old Len= 123;New Len= 131 module=CHAIN
2025-10-12|01:37:05.660 [ERROR] Failed to process block:hash=0x2da50a24fb415b65d2e4bcc9de6100a0ce8051c53d3a1a019ccc29dede4433f5 err=already have block 0x2da50a24fb415b65d2e4bcc9de6100a0ce8051c53d3a1a019ccc29dede4433f5 module=SYNC processID=3636
2025-10-12|01:37:05.660 [ERROR] Failed to process block:hash=0xd721fe69bbd781dbf335f774bef53b6b3083145488b90dc71c24382a0b7564d0 err=already have block 0xd721fe69bbd781dbf335f774bef53b6b3083145488b90dc71c24382a0b7564d0 module=SYNC processID=3636
2025-10-12|01:37:05.663 [ERROR] Failed to process block:hash=0x9664afe61acd5c70d6f2ee29113f31ed91aaae80e28d24f5beea6d73189a2605 err=already have block 0x9664afe61acd5c70d6f2ee29113f31ed91aaae80e28d24f5beea6d73189a2605 module=SYNC processID=3636
2025-10-12|01:37:05.664 [INFO ] The sync of graph state has ended module=SYNC spend=1s processID=3636
2025-10-12|01:37:05.664 [INFO ] You're up to date now. module=SYNC
2025-10-12|01:37:05.664 [INFO ] update BdagPool snapshot module=BDAG size=0
2025-10-12|01:37:06.009 [INFO ] Start DAG REORGANIZE: Block 0x43b15b8d67f2e0e50fbd396c9b71bdb3a6dbf6b32403f27e737a811a671dcf82 is causing a reorganize. module=CHAIN
2025-10-12|01:37:06.009 [INFO ] Try to rewind module=BDAG cur.number=9884053 cur.hash=0x31a0270d4fd01a5ca1dccb652bcbdfececab46f468a15fd3dd20969ebd6f3ce7 target.evm.root=34ebd6..9f325a target.evm.number=9883922 target.evm.hash=a0ddce..df217c
2025-10-12|01:37:06.009 [WARN ] Rewinding blockchain to block target=9,883,922
2025-10-12|01:37:06.009 [ERROR] Zero state root hash!
2025-10-12|01:37:06.009 [INFO ] Rewound to block with state number=9,884,052 hash=d53f3b..738284
...
2025-10-12|01:37:06.078 [INFO ] Rewound to block with state number=9,883,926 hash=055118..e42f82
2025-10-12|01:37:06.079 [ERROR] Zero state root hash!
2025-10-12|01:37:06.079 [INFO ] Rewound to block with state number=9,883,925 hash=db2ada..18dbd1
2025-10-12|01:37:06.079 [ERROR] Zero state root hash!
2025-10-12|01:37:06.079 [CRIT ] Failed to rollback state err="state is unrecoverable"
2025-10-12|01:37:06.162 [ERROR] watchExecuted: subscription error err="websocket: close 1006 (abnormal closure): unexpected EOF"
2025-10-12|01:37:06.163 [INFO ] watchExecuted: reconnecting in 2s
2025-10-12|01:37:06.240 [WARN ] liveness probe failed url=http://127.0.0.1:6061/healthz err="Get "http://127.0.0.1:6061/healthz\": dial tcp 127.0.0.1:6061: connect: connection refused"
2025-10-12|01:37:08.163 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"
2025-10-12|01:37:10.164 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"
2025-10-12|01:37:12.165 [ERROR] watchExecuted: dial WS failed err="dial tcp 127.0.0.1:18546: connect: connection refused"
root@5c362fad6687:/bdag# bdag --metrics --testnet
2025-10-12|06:40:41.673 [INFO ] System info BDAG Version=1.0.0 Go version=go1.22.12
2025-10-12|06:40:41.673 [INFO ] System info Home dir=.
2025-10-12|06:40:41.673 [INFO ] Using pebble as the backing database module=RAWDB
2025-10-12|06:40:41.752 [INFO ] Opened ancient database module=RAWDB database=data/testnet/BdagChain/ancient/chain readonly=false
2025-10-12|06:40:41.753 [INFO ] transaction index is enabled module=INDEX
2025-10-12|06:40:41.753 [INFO ] anticone size:43 module=DAG
2025-10-12|06:40:41.753 [INFO ] bdag chain module=BDAG version=bdagvm-v0.0.2
2025-10-12|06:40:41.762 [INFO ] New local node record seq=1,760,251,241,761 id=bef4527c57493bd5 ip=127.0.0.1 udp=8538 tcp=0
2025-10-12|06:40:41.765 [INFO ] Prepare bdageth on NetWork(1043)...
2025-10-12|06:40:41.765 [INFO ] Enabling metrics collection
2025-10-12|06:40:41.767 [INFO ] Maximum peer count ETH=0 total=0
2025-10-12|06:40:41.768 [INFO ] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
2025-10-12|06:40:41.770 [INFO ] Set global gas cap cap=50,000,000
2025-10-12|06:40:41.770 [INFO ] Initializing the KZG library backend=gokzg
2025-10-12|06:40:41.782 [INFO ] Allocated trie memory caches clean=154.00MiB dirty=256.00MiB
2025-10-12|06:40:41.782 [INFO ] Using pebble as the backing database
2025-10-12|06:40:41.782 [INFO ] Allocated cache and file handles database=/bdag/data/testnet/bdageth/chaindata cache=512.00MiB handles=536,870,908
2025-10-12|06:40:42.474 [INFO ] Opened ancient database database=/bdag/data/testnet/bdageth/chaindata/ancient/chain readonly=false
2025-10-12|06:40:42.475 [INFO ] State scheme set to already existing scheme=path
2025-10-12|06:40:42.475 [INFO ] Initialising Ethereum protocol network=1043 dbversion=8
2025-10-12|06:40:42.475 [INFO ] Failed to load journal, discard it err="journal not found"
2025-10-12|06:40:42.486 [INFO ] Opened ancient database database=/bdag/data/testnet/bdageth/chaindata/ancient/state readonly=false
2025-10-12|06:40:42.528 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=storage.index items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.531 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=account.data items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.535 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=storage.data items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.537 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=history.meta items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.541 [WARN ] Truncating freezer table database=/bdag/data/testnet/bdageth/chaindata/ancient/state table=account.index items=9,883,926 limit=9,818,029
2025-10-12|06:40:42.545 [WARN ] Truncated extra state histories number=65897
2025-10-12|06:40:42.545 [INFO ]
2025-10-12|06:40:42.545 [INFO ] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2025-10-12|06:40:42.545 [INFO ] Chain ID: 1043 (bdag-test)
2025-10-12|06:40:42.545 [INFO ] Consensus: Ethash (proof-of-work)
2025-10-12|06:40:42.545 [INFO ]
2025-10-12|06:40:42.545 [INFO ] Pre-Merge hard forks (block based):
2025-10-12|06:40:42.545 [INFO ] - Homestead: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
2025-10-12|06:40:42.545 [INFO ] - DAO Fork: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
2025-10-12|06:40:42.545 [INFO ] - Tangerine Whistle (EIP 150): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
2025-10-12|06:40:42.545 [INFO ] - Spurious Dragon/1 (EIP 155): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2025-10-12|06:40:42.545 [INFO ] - Spurious Dragon/2 (EIP 158): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2025-10-12|06:40:42.545 [INFO ] - Byzantium: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
2025-10-12|06:40:42.545 [INFO ] - Constantinople: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
2025-10-12|06:40:42.545 [INFO ] - Petersburg: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
2025-10-12|06:40:42.545 [INFO ] - Istanbul: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
2025-10-12|06:40:42.546 [INFO ] - Muir Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
2025-10-12|06:40:42.546 [INFO ] - Berlin: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
2025-10-12|06:40:42.546 [INFO ] - London: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] The Merge is not yet available for this network!
2025-10-12|06:40:42.546 [INFO ] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] Post-Merge hard forks (timestamp based):
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2025-10-12|06:40:42.546 [INFO ]
2025-10-12|06:40:42.546 [INFO ] Loaded most recent local block number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m22s
2025-10-12|06:40:42.546 [WARN ] Head state missing, repairing number=9,883,925 hash=db2ada..18dbd1 snaproot=40388e..4191e8
2025-10-12|06:40:43.034 [INFO ] Rewound to block with state number=9,818,028 hash=b8682b..10bcc3
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local header number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m23s
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local block number=9,818,028 hash=b8682b..10bcc3 td=9,818,028 age=7h45m14s
2025-10-12|06:40:43.040 [INFO ] Loaded most recent local snap block number=9,883,925 hash=db2ada..18dbd1 td=9,883,925 age=5h4m23s
2025-10-12|06:40:43.041 [WARN ] Enabling snapshot recovery chainhead=9,818,028 diskbase=9,831,120
2025-10-12|06:40:43.042 [WARN ] Loaded snapshot journal diskroot=40388e..4191e8 diffs=unmatched
2025-10-12|06:40:43.042 [WARN ] Snapshot is not continuous with chain snaproot=40388e..4191e8 chainroot=ae45bf..8c289e
2025-10-12|06:40:43.042 [INFO ] Initialized transaction indexer range="last 2350000 blocks"
2025-10-12|06:40:43.042 [INFO ] Setting new local account address=0x250d49eEe9ed2ec5Fdc59d9B0f4Eb8202B03f212
2025-10-12|06:40:43.042 [INFO ] Loaded local transaction journal transactions=1 dropped=0
2025-10-12|06:40:43.042 [INFO ] Regenerated local transaction journal transactions=1 accounts=1
2025-10-12|06:40:43.053 [WARN ] Switch sync mode from snap sync to full sync reason="snap sync complete"
2025-10-12|06:40:43.054 [INFO ] Gasprice oracle is ignoring threshold set threshold=2
2025-10-12|06:40:43.055 [WARN ] Old unclean shutdowns found count=16
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T00:27:48+0000 age=6d6h12m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T08:45:04+0000 age=5d21h55m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-06T18:12:01+0000 age=5d12h28m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-07T04:49:21+0000 age=5d1h51m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T00:21:18+0000 age=4d6h19m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T02:57:48+0000 age=4d3h42m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-08T06:41:12+0000 age=3d23h59m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-09T03:37:23+0000 age=3d3h3m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-10T04:17:17+0000 age=2d2h23m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-10T16:42:24+0000 age=1d13h58m
2025-10-12|06:40:43.055 [WARN ] Unclean shutdown detected booted=2025-10-12T01:34:18+0000 age=5h6m25s
2025-10-12|06:40:43.056 [INFO ] bdag pool init... module=BDAG
2025-10-12|06:40:43.056 [INFO ] Skipping genesis transaction re-indexing for empty block in EVM model module=CDB
2025-10-12|06:40:43.056 [INFO ] Loading dag ... module=CHAIN
2025-10-12|06:40:43.056 [WARN ] get block module=DAG error=empty blockID=10321229
2025-10-12|06:40:43.056 [INFO ] Shutdown complete
2025-10-12|06:40:43.056 [ERROR] The dag data was damaged (Can't find tip:10321229). you can cleanup your block data base by '--cleanup'.