[RPC] Allow changing L1 synced height via authenticated RPC

[greged93]

Add an authenticated RPC endpoint on the rollup node that allows for the admin to update the L1 synced height.

[Thegaram]

What does "authenticated" mean in this context?

I think this term is usually used for the engine API (that uses JWT authentication), but the admin_ namespace is not authenticated in geth, so one must take care not to expose it externally.

[greged93]

the admin_ namespace is not authenticated in geth, so one must take care not to expose it externally.

does it just run on another port on localhost then?

[Thegaram]

Same port actually. For public-facing APIs, we have a gateway that only exposes certain methods. I suppose most RPC providers do this.

That said, we do not need to follow l2geth's approach exactly.

[greged93]

Same port actually. For public-facing APIs, we have a gateway that only exposes certain methods. I suppose most RPC providers do this.

I suggested the authenticated method in the issue because I was not aware we could do this, which would probably be a much simpler solution. How did you configure this? Just manually in the gateway or did you have a config file with the list of exposed methods?

[georgehao]

I think geth don't expose admin_* api is because it can be reached through geth console. but reth don't have a console as geth. So I think authenticated RPC endpoint is reasonable

[Thegaram]

How did you configure this? Just manually in the gateway or did you have a config file with the list of exposed methods?

For public RPC, we use https://github.com/scroll-tech/rpc-gateway which only allows a predefined set of RPCs. I believe geth's model is: (1) If your node is externally facing, do not enable sensitive endpoints like admin and debug, (2) If you do, make sure to maintain strict control to the node (e.g. via a proxy).

[frisitano]

In what context would we expect this to be used? Is this a mitigation for potentially missed events? Can we discuss more details around the desired behaviour and explain its use in a specific context? If we are modifying the L1 synced height, then should we purge all data associated with L1 blocks greater than the revert height? Additionally, would we want to revert the L2 chain head if the chain includes L1 messages from these blocks or should we instead transition to optimistic mode? We could have the revert L2 / switch to optimistic as an argument to the rpc but it would be good to understand the context in which this may be used. cc: @Thegaram @greged93 @jonastheis

[Thegaram]

Is this a mitigation for potentially missed events?

Yes.

In l2geth we have admin_setL1MessageSyncedL1Height and admin_setRollupEventSyncedL1Height.

This is very useful in the case that the node missed some events (due to a bug, or a misbehaving L1 node). If re-syncing data is idempotent then we can always safely reset sync height, and this recovery option is always available. (Idempotency in l2geth: If synced message is already in DB, we do not reset the highest seen queue index either.)

If we are modifying the L1 synced height, then should we purge all data associated with L1 blocks greater than the revert height?

Ideally only if we sync data that's different from previously processed data. This can only happen if we relay unfinalized data from L1, or if the L1 node is returning incorrect data (unlikely).

Additionally, would we want to revert the L2 chain head if the chain includes L1 messages from these blocks or should we instead transition to optimistic mode?

Same, only revert/reorg if the re-synced data does not match previously processed data.

[jonastheis]

The main issue here seems to be "the node missed some events" and the best solution to that is introducing a self-healing mechanism to address the issue without needing user intervention. As such I'd say the API is secondary and we can consider this after we implement the mechanism to address the fundamental underlying issue automatically.

Self-healing mechanism

For this to work we need to be able to detect a missing event.

• L1 message missing
• batch commit event missing
• what happens if a batch revert event is missing?
• a missed batch finalization event should be okay due to the fact that a batch finalization always implicitly finalizes all batches before.

iiuc there's currently a missing event detection in the business logic of the respective component to detect a missing event but no automatic recovery implemented. However, recovery is not trivial if we only detect missing events at component level because there might be unexpected side effects. Specifically, there might be events queried during the same L1 watcher batch where e.g. a L1 message is missing but a batch commit event is received. The L1 watcher would then pass the batch commit event on to be handled by the derivation pipeline but only later the missing L1 message would be detected. This could result in complicated revert/replay logic when the same block range is queried again.

Proposed solution:
A conceptual elegant solution would be to filter events at the source, ie in L1 watcher by some rules. Only if a batch of events queried from L1 passes the filter the events would be propagated further. Otherwise, we go back to the latest "good" state which is the last time events existed and passed the filter and query L1 blocks from there.

This should effectively address the issue "the node missed some events" at the source and not introduce any complexities of needing to handle revert/replay of events (since the components didn't even know about the events).

The question now would be how we would implement this and how much effort it would be.

[Thegaram]

filter events at the source, ie in L1 watcher by some rules. Only if a batch of events queried from L1 passes the filter the events would be propagated further.

Can you provide an example here?

Do you mean rules like "L1 message is only valid if the previous L1 message is known", and "batch commit event is only valid if the L1 messages contained in it are known"?

But do we have all the necessary information available in the watcher? I'm thinking, batch commits might need to pass through the derivation pipeline first.