Feat/optimize rpc#718
Draft
konradkonrad wants to merge 9 commits into
Draft
Conversation
Context is issue #698
This adds (generated) integration tests and a `mise` test task for ensuring the changed batching behavior when responding to event based decryption triggers. Context is #698
ActivationBlockNumber can not be negative, so no overflow risk here.
Also apply `black` on `test-event-decryption`.
- deploy test helper in `deploy` - untangled test and setup: - explicit `submit-event-registration` task - explicit `trigger-event` task - explicit `wait-for-event-decryption-key`
- New() precomputes the union of contract addresses, the union of event topic0 hashes, and a (address, topic) -> *EventType dispatch table. - syncAllInRange() now issues a single FilterLogs covering every registered (address, topic), then dispatches each returned log to its EventType via the precomputed table. Logs that match the query but have no registered handler (cross-product hits) are skipped. - Removed the per-event-type fan-out (syncSingleInRange) along with the sync.Mutex and golang.org/x/sync/errgroup dependencies. For the snapshot keyper (the only current caller via chainobserver), this drops per-block eth_getLogs requests from N (one per event type) to 1. Behavior is preserved: the same logs reach Next() in the same block + log-index order.
keyperimpl/shutterservice/eventprocessor.go — new interface
- FilterCriteria struct carries Addresses []common.Address and Topics0 []common.Hash (empty
Topics0 means "any topic" for that processor).
- EventProcessor interface replaces the old FetchEvents(ctx, start, end) with
FilterCriteria(ctx, start, end) + ParseEvents(ctx, start, end, logs).
keyperimpl/shutterservice/multieventsyncer.go — combined fetch
- gatherFilterCriteria asks each processor for its criteria, builds the unioned address list
and topic[0] list, captures per-processor scopes for routing.
- syncRange now issues at most one FilterLogs(addresses, [topic0Union]) per range (skipped
entirely if no processor wants any addresses), then routes the matching subset of logs to
each processor's ParseEvents. If any processor declines to constrain topic[0], the combined
query drops topic filtering.
keyperimpl/shutterservice/eventtriggerregisteredprocessor.go
- Constructor now takes the contract address and parses the bindings' ABI to extract the
EventTriggerRegistered topic. Returns an error from construction.
- FilterCriteria returns {[address], [topic]}.
- ParseEvents calls Contract.ParseEventTriggerRegistered(log) (abigen helper) per log.
keyperimpl/shutterservice/triggerprocessor.go
- New activeTriggers helper reads + parses active trigger definitions from the DB.
- FilterCriteria unions the addresses and topic[0] hashes across all active triggers; sets
anyTopic when a trigger leaves topic[0] unconstrained.
- ParseEvents iterates each trigger over the pre-filtered logs and emits TriggerEvents for
ones that pass def.Match and aren't expired.
keyperimpl/shutterservice/keyper.go
- Updated NewEventTriggerRegisteredEventProcessor call to pass the address and handle the new
error return.
Steady-state RPC count per block range with N active triggers drops from 1
(FilterEventTriggerRegistered) + N (per-trigger FilterLogs) to 1 combined FilterLogs. The
HeaderByNumber(end) is still there (that's S3, not in scope).
Note: there's one DB cost trade-off worth flagging — GetActiveEventTriggerRegisteredEvents is
now queried twice per range inside TriggerProcessor (once in FilterCriteria, once in
ParseEvents). S8 in the doc covers the cache that would eliminate this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Select optimizations to reduce RPC call count, based on analysis from #717:
S2 optimization
keyperimpl/shutterservice/eventprocessor.go— new interface-
FilterCriteriastruct carriesAddresses []common.AddressandTopics0 []common.Hash(emptyTopics0means "any topic" for that processor).-
EventProcessorinterface replaces the oldFetchEvents(ctx, start, end)withFilterCriteria(ctx, start, end)+ParseEvents(ctx, start, end, logs).keyperimpl/shutterservice/multieventsyncer.go— combined fetch-
gatherFilterCriteriaasks each processor for its criteria, builds the unionedaddresslist andtopic[0]list, captures per-processor scopes for routing.-
syncRangenow issues at most oneFilterLogs(addresses, [topic0Union])per range (skipped entirely if no processor wants any addresses), then routes the matching subset of logs to each processor'sParseEvents. If any processor declines to constraintopic[0], the combined query drops topic filtering.keyperimpl/shutterservice/eventtriggerregisteredprocessor.go- Constructor now takes the contract address and parses the bindings' ABI to extract the
EventTriggerRegisteredtopic. Returns an error from construction.-
FilterCriteriareturns{[address], [topic]}.-
ParseEventscallsContract.ParseEventTriggerRegistered(log)(abigen helper) per log.keyperimpl/shutterservice/triggerprocessor.go- New
activeTriggershelper reads + parses active trigger definitions from the DB.-
FilterCriteriaunions theaddressesandtopic[0]hashes across all active triggers; setsanyTopicwhen a trigger leavestopic[0]unconstrained.-
ParseEventsiterates each trigger over the pre-filtered logs and emitsTriggerEventsfor ones that passdef.Matchand aren't expired.keyperimpl/shutterservice/keyper.go- Updated
NewEventTriggerRegisteredEventProcessorcall to pass the address and handle the new error return.Steady-state RPC count per block range with N active triggers drops from 1 (
FilterEventTriggerRegistered) + N (per-triggerFilterLogs) to 1 combinedFilterLogs. TheHeaderByNumber(end)is still there (that's S3, not in scope).Note: there's one DB cost trade-off worth flagging -
GetActiveEventTriggerRegisteredEventsis now queried twice per range insideTriggerProcessor(once inFilterCriteria, once inParseEvents). S8 in the doc covers the cache that would eliminate this.B4 optimization
New()precomputes the union of contract addresses, the union of eventtopic0hashes, and an(address, topic) -> *EventTypedispatch table.syncAllInRange()now issues a singleFilterLogscovering every registered(address, topic), then dispatches each returned log to itsEventTypevia the precomputed table. Logs that match the query but have no registered handler (cross-product hits) are skipped.syncSingleInRange) along with thesync.Mutexandgolang.org/x/sync/errgroupdependencies.For the snapshot keyper (the only current caller via chainobserver), this drops per-block
eth_getLogsrequests from N (one per event type) to 1. Behavior is preserved: the same logs reachNext()in the same block + log-index order.