Checkpointz stopped working for Gnosis mainnet

[zheli]

Gnosis mainnet just had their Shapella upgrade on Aug 1st, 2023 and it seems that has broken all the gnosis mainnet checkpoint services. Would be great if you can help us to figure out what's wrong.

The error in checkpointz:

level=error msg="Failed to check for new serving checkpoint" error="block root does not match" module=beacon/default

The error in lighthouse:

Remote BN does not support EIP-4881 fast deposit sync

Thanks!

[zheli]

More logs from checkpointz:

time="2023-08-02T12:31:24Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:31:26Z" level=info msg="Fetching bundle from node Gateway FM 1 with root 0xb865aadb1a81ff4fd03f69670d1de732ba4cbf0b29c1160cc49ed741d51c449d" module=beacon/default
time="2023-08-02T12:31:26Z" level=error msg="Failed to check for new serving checkpoint" error="block root does not match" module=beacon/default
time="2023-08-02T12:31:26Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:31:26Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:31:28Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:31:28Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:31:30Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:31:46Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:31:47Z" level=info msg="New finalized head checkpoint" epoch=649825 module=beacon/default root=0x53cab6f7d5a54b5d4c11319b0c7add06f432444d1245da376f2126a787b3358d
time="2023-08-02T12:31:47Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:31:47Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:31:50Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:31:50Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:31:53Z" level=debug msg="Added block" block_root=0x4975d9bc80f95cf694b699eca64ce7404b249bbcd4b45fdeafb1a91ad0c3c616 component=beacon/store/block expires_at="2023-08-16 12:31:53.649236943 +0000 UTC m=+1209731.417621044" slot=10397184 state_root=0x798c64c8cd6cf8194721031e3513d2de926571016fd06df5e77503e83fd2cb7b
time="2023-08-02T12:31:53Z" level=info msg="Downloaded and stored block for slot 10397184" module=beacon/default root=0x4975d9bc80f95cf694b699eca64ce7404b249bbcd4b45fdeafb1a91ad0c3c616 slot=10397184 state_root=0x798c64c8cd6cf8194721031e3513d2de926571016fd06df5e77503e83fd2cb7b
time="2023-08-02T12:31:57Z" level=info msg="Fetching bundle from node Gateway FM 2 with root 0x53cab6f7d5a54b5d4c11319b0c7add06f432444d1245da376f2126a787b3358d" module=beacon/default
time="2023-08-02T12:31:57Z" level=error msg="Failed to check for new serving checkpoint" error="block root does not match" module=beacon/default
time="2023-08-02T12:32:00Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:32:15Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:32:28Z" level=info msg="Fetching bundle from node Gateway FM 3 with root 0x53cab6f7d5a54b5d4c11319b0c7add06f432444d1245da376f2126a787b3358d" module=beacon/default
time="2023-08-02T12:32:28Z" level=error msg="Failed to check for new serving checkpoint" error="block root does not match" module=beacon/default
time="2023-08-02T12:32:30Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:32:45Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:32:59Z" level=info msg="Fetching bundle from node Gateway FM 1 with root 0x53cab6f7d5a54b5d4c11319b0c7add06f432444d1245da376f2126a787b3358d" module=beacon/default
time="2023-08-02T12:32:59Z" level=error msg="Failed to check for new serving checkpoint" error="block root does not match" module=beacon/default
time="2023-08-02T12:33:00Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready
time="2023-08-02T12:33:00Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:33:00Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/beacon/slots
time="2023-08-02T12:33:05Z" level=debug msg="Handling request" accept="*/*" content_type=application/json method=GET module=api path=/checkpointz/v1/status
time="2023-08-02T12:33:07Z" level=info msg="New finalized head checkpoint" epoch=649826 module=beacon/default root=0x08bed0c9b0cd5ef51bf2ede844c96d58212294444fd1fbe53238d19966e5f722
time="2023-08-02T12:33:08Z" level=debug msg="Added block" block_root=0xcd1516f2befe2cf6c088c1363753b47e5e8edb9d6951ca899c2603c2d2dc5cd5 component=beacon/store/block expires_at="2023-08-16 12:33:08.76897931 +0000 UTC m=+1209806.537363410" slot=10397200 state_root=0x1bee5ce1d6036cabd3d3e49b94c63c773eaf1c7f0a68072f8115a87f071249e8
time="2023-08-02T12:33:08Z" level=info msg="Downloaded and stored block for slot 10397200" module=beacon/default root=0xcd1516f2befe2cf6c088c1363753b47e5e8edb9d6951ca899c2603c2d2dc5cd5 slot=10397200 state_root=0x1bee5ce1d6036cabd3d3e49b94c63c773eaf1c7f0a68072f8115a87f071249e8
time="2023-08-02T12:33:15Z" level=debug msg="Handling request" accept= content_type=application/json method=GET module=api path=/checkpointz/v1/ready

[emlautarom1]

Also having issues with Teku: on startup, when adding the --initial-state=https://checkpoint.gnosischain.com/eth/v2/debug/beacon/states/finalized option, it crashes with an exception.