move pebble checkpoints off of the execution goroutine - #4039
move pebble checkpoints off of the execution goroutine#4039cody-littley wants to merge 1 commit into
Conversation
PR SummaryHigh Risk Overview Backpressure and defaults: New API and lifecycle: Public Tests and tooling: Integration tests wait for async state-store application and snapshot publication; Reviewed by Cursor Bugbot for commit 96c22ca. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4039 +/- ##
==========================================
- Coverage 61.22% 60.53% -0.70%
==========================================
Files 2153 2079 -74
Lines 188393 180394 -7999
==========================================
- Hits 115351 109205 -6146
+ Misses 62298 61056 -1242
+ Partials 10744 10133 -611
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 96c22ca. Configure here.
| // reconciles whatever the caller's halt leaves behind. | ||
| if s.config.SnapshotInterval > 0 && version%int64(s.config.SnapshotInterval) == 0 { | ||
| s.phaseTimer.SetPhase("commit_write_snapshot") | ||
| if err := s.WriteSnapshot(""); err != nil { |
There was a problem hiding this comment.
Export races with async snapshot pruning
High Severity
Commit now returns after Offer, so a snapshot interval height is not on disk yet. Exporter still clones a snapshot directory without FlushSnapshots, while publishSnapshot can prune that directory mid-copy. State-sync at an interval height may replay a full WAL gap or copy a directory the writer is deleting.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 96c22ca. Configure here.
| return | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Close may process queued snapshots
Medium Severity
After an in-flight checkpoint, run loops back into a select on ctx.Done() and messages. If both are ready, Go picks at random, so Close can write more queued snapshots instead of discarding them. Shutdown then waits on extra checkpoints that the contract says are dropped.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 96c22ca. Configure here.
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Moving FlatKV pebble checkpointing onto a dedicated SnapshotWriter goroutine is well-structured: reservation ownership is handed off cleanly at every path (written/declined/failed/discarded), failures are latched and re-surfaced on the commit path, and teardown drains an in-flight checkpoint before the databases close. Two non-blocking issues: the new max-snapshot-lag-blocks key is declared configurable but never read, and the 256× increase in MaxUnflushedVersions raises an in-memory backlog that has no byte-based bound.
Findings: 0 blocking | 2 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- None at the file/PR level.
- 2 suggestion(s)/nit(s) flagged inline on specific lines.
| // reach disk until it completes, and each one is retained in memory meanwhile. This bounds how far | ||
| // that can run, trading a pause in block production for the memory the backlog would otherwise | ||
| // consume. It bounds blocks rather than bytes, so it mitigates exhaustion rather than preventing it. | ||
| MaxSnapshotLagBlocks uint32 `mapstructure:"max-snapshot-lag-blocks"` |
There was a problem hiding this comment.
[suggestion] The mapstructure:"max-snapshot-lag-blocks" tag is inert: neither sei-cosmos/server/config.GetConfig (which explicitly reads five other state-commit.flatkv.* keys — fsync, async-write-buffer, snapshot-interval, snapshot-keep-recent, enable-read-write-metrics) nor app/parseSCConfigs reads this key, and the Viper path ignores unknown TOML keys. An operator who sets state-commit.flatkv.max-snapshot-lag-blocks in app.toml gets silence, and the queue stays pinned at 512.
This knob is the whole of the writer's backpressure and the one lever an operator has when a checkpoint outruns block production, so it is the flatkv field most worth wiring rather than least. Suggest adding the guarded read alongside the other four in GetConfig, and a row in sei-cosmos/server/config/config_fuzz_test.go ({Key: "state-commit.flatkv.max-snapshot-lag-blocks", Path: "StateCommit.FlatKVConfig.MaxSnapshotLagBlocks", Set: ...}) per testutil/configtest/AGENTS.md. If it is deliberately not operator-tunable for now, dropping the tag (as ExternalPruning does with mapstructure:"-") records that decision instead of implying one that does not hold.
| MetricsEnabled: true, | ||
| MetricsScrapeIntervalSeconds: 10, | ||
| MaxUnflushedVersions: 4, | ||
| MaxUnflushedVersions: 1024, |
There was a problem hiding this comment.
[suggestion] Raising the default from 4 to 1024 applies to all four FlatKV view managers, and this is the only bound on that backlog: MaxSize above is documented as constraining "only the DB read-cache, not view data, since views cannot be freed without compromising consistency semantics". So the cap is a version count with no byte-based counterpart — 1024 block diffs of arbitrary size may now be resident per manager, where previously it was 4.
The increase is required by the design (with 4, Commit would stall inside view-manager backpressure for the whole checkpoint, defeating the point of the PR), and MaxSnapshotLagBlocks caps the snapshot-driven backlog at ~513. But the new value also governs the unrelated case this field was written for — Pebble being the bottleneck — where nothing bounds the accumulation at 512, and a slow-flush episode can now hold 256× as much in memory before backpressure engages.
Worth recording the reasoning here (why 1024 rather than something just above MaxSnapshotLagBlocks) and noting the relationship between the two knobs, since they are set in different packages and only stay consistent by hand.
| MetricsEnabled: true, | ||
| MetricsScrapeIntervalSeconds: 10, | ||
| MaxUnflushedVersions: 4, | ||
| MaxUnflushedVersions: 1024, |
There was a problem hiding this comment.
any OOM risk here to MaxUnflushedVersions to 1024?
| // Step 5: Offer the block to the snapshot writer, which decides whether it becomes a snapshot and, | ||
| // if so, writes it on its own goroutine. Periodic snapshots are what keep the WAL bounded and | ||
| // restarts fast. | ||
| if s.snapshotWriter != nil { |
There was a problem hiding this comment.
Offer now reports the writer's latched failure, so the error names the current height, not the height whose snapshot failed. The halt is also one block late. Both are acceptable if nothing downstream reads that height for recovery. Please confirm, and consider naming the failed height in the latched error so the log points at the real block.
| // cadence is outrunning. | ||
| w.phaseTimer.SetPhase("idle") | ||
| select { | ||
| case <-w.ctx.Done(): |
There was a problem hiding this comment.
select randomly chooses among ready cases, so shutdown competes with the queue instead of taking priority. Once stop() is called, w.ctx.Done() remains ready, causing races on both sides of the channel.
Here, a queued message may win, causing Close to wait for another snapshot and breaking its guarantee that queued work is discarded.
In enqueue at L166, a send may win if the queue has room, so Offer returns nil even though the writer may exit without processing it. Commit then reports success for a snapshot that may never be written.
Both sides need to be fixed: one prevents extra work during shutdown; the other prevents false success after shutdown.
| FlatKVConfig.AccountStoreConfig.MetricsEnabled = bool(true) | ||
| FlatKVConfig.AccountStoreConfig.MetricsScrapeIntervalSeconds = float64(10) | ||
| FlatKVConfig.AccountStoreConfig.MaxUnflushedVersions = uint64(4) | ||
| FlatKVConfig.AccountStoreConfig.MaxUnflushedVersions = uint64(1024) |
There was a problem hiding this comment.
This seems to be a pretty big jump? Why do we need that big of backlog?
3 participants
Describe your changes and provide context
Moves pebble checkpointing (what we call snapshots) off of the transaction execution goroutine