Skip to content

Bound /store/*/subspace ABCI queries - #4009

Open
amir-deris wants to merge 6 commits into
mainfrom
amir/plt-799-abci-query-exhaustion-dos
Open

Bound /store/*/subspace ABCI queries #4009
amir-deris wants to merge 6 commits into
mainfrom
amir/plt-799-abci-query-exhaustion-dos

Conversation

@amir-deris

@amir-deris amir-deris commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes PLT-799. The /store/{name}/subspace ABCI path on storev2 SS nodes iterated an entire KV prefix into memory with no empty-data guard, no result cap, and no concurrency limit.

This PR makes /subspace fail closed on unbounded work:

  • Reject empty prefix in storev2/query.ScanSubspace before any iteration (covers state and commitment paths).
  • Cap during iteration via the shared storev2/query helper: stop once pair count or accumulated key+value bytes exceed configured limits (defaults: 1,000 pairs / 4 MiB). Returns typed store.ErrSubspaceCapExceeded — no silent truncation.
  • Context-aware iteration via IteratorOn so SS MVCC skip loops honor request cancellation and deadlines (not just between yielded pairs).
  • Concurrency semaphore on the SS fast path (subspaceQuerySem, separate from histProofSem): saturated callers get ErrConflict, same as historical proof.
  • OTel metric subspace_query_rejected{reason=semaphore|cap_exceeded} for operator visibility.

Query-only change — no AppHash or chain-upgrade impact.

Operator note: wide /subspace queries that previously returned large result sets may now fail with ErrSubspaceCapExceeded. Narrow prefixes or raise sc-subspace-max-pairs / sc-subspace-max-bytes before upgrading if indexers or tooling depend on wide scans.

New config ([state-commit])

Key Default Notes
sc-subspace-query-max-inflight 2 <=0 resolves to default
sc-subspace-max-pairs 1,000 <=0 resolves to default; no unlimited setting
sc-subspace-max-bytes 4 MiB (4,194,304) <=0 resolves to default; no unlimited setting

Wired through parseSCConfigs, app.toml template, and config characterization goldens.

Test plan

  • go test ./sei-cosmos/storev2/query/... — pair/byte caps, ctx cancel, typed error + ABCI code preservation
  • go test ./sei-cosmos/storev2/commitment/... — empty prefix rejected, narrow prefix succeeds, /key unaffected
  • go test ./sei-cosmos/storev2/rootmulti/... — semaphore saturation, integration paths, empty prefix
  • go test ./app/ -run 'TestDefaultsMatchTheRecordedValues|TestKeyNamesMatchTheRecordedNames|FuzzParseSCConfigs'
  • go test ./sei-db/config/...

Reject empty prefixes, cap pair/byte accumulation during iteration, and limit concurrent SS fast-path scans so unauthenticated callers cannot OOM a node with wide subspace queries.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris amir-deris self-assigned this Aug 25, 2026
@amir-deris amir-deris changed the title Bound /subspace ABCI queries to prevent memory-exhaustion DoS (PLT-799). Bound /store/*/subspace ABCI queries against memory-exhaustion DoS (PLT-799) Aug 25, 2026
@amir-deris
amir-deris marked this pull request as ready for review August 25, 2026 15:22
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedAug 26, 2026, 5:01 PM

@cursor

cursor Bot commented Aug 25, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Query-only hardening with a breaking change for clients that depend on large /subspace results; no consensus or AppHash impact, but misconfigured indexers may fail after upgrade.

Overview
Caps /store/*/subspace ABCI queries that previously materialized an entire KV prefix in memory with no bounds. Wide scans now fail with subspace result exceeds limit once they would exceed defaults of 1,000 pairs or 4 MiB of key+value bytes; empty prefixes are rejected. /key queries are unchanged.

Implementation centralizes iteration in storev2/query.ScanSubspace, which uses context-aware IteratorOn (cancellation/deadlines) and returns typed ErrSubspaceCapExceeded instead of panicking on errors. State-commit and SS query paths (commitment.Store, state.Store, rootmulti) all receive shared limits from new [state-commit] knobs: sc-subspace-max-pairs, sc-subspace-max-bytes, and sc-subspace-query-max-inflight (default 2 concurrent unproven /subspace on the SS fast path; saturated callers get subspace query busy). Rejections are counted in subspace_query_rejected (reason=semaphore|cap_exceeded).

Operator impact: indexers or tooling that relied on wide /subspace scans must narrow prefixes, shard by sub-prefix, or raise the config caps before upgrade; <= 0 resolves to defaults with no unlimited mode.

Reviewed by Cursor Bugbot for commit 260ccf0. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread sei-cosmos/storev2/query/subspace.go
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.28%. Comparing base (b117536) to head (260ccf0).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4009      +/-   ##
==========================================
- Coverage   61.34%   60.28%   -1.06%     
==========================================
  Files        2163     2059     -104     
  Lines      188757   176566   -12191     
==========================================
- Hits       115792   106445    -9347     
+ Misses      62256    60410    -1846     
+ Partials    10709     9711     -998     
Flag Coverage Δ
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
app/seidb.go 90.90% <ø> (ø)
sei-cosmos/storev2/commitment/store.go 56.47% <ø> (+4.42%) ⬆️
sei-cosmos/storev2/rootmulti/metrics.go 50.00% <ø> (ø)
sei-cosmos/storev2/rootmulti/store.go 68.89% <ø> (-0.56%) ⬇️
sei-cosmos/storev2/state/store.go 31.81% <ø> (+1.43%) ⬆️
sei-db/config/sc_config.go 100.00% <ø> (ø)

... and 142 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-tested DoS hardening of the storev2 /subspace ABCI path: pair/byte caps, an empty-prefix rejection, an SS-path semaphore, and complete config-characterization wiring (flags, defaults, app.toml template, both goldens, and the fuzz seed re-indexing all line up). Three non-blocking issues: the scan ignores the repo's context-aware iterator helper, the empty-prefix guard is duplicated at callers instead of the shared choke point, and the 0-means-default cap semantics are undocumented and contradict the neighbouring rate-limit key.

Findings: 0 blocking | 5 non-blocking | 3 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • [suggestion] The new in-flight semaphore only covers the SS fast path. Since RequireProof("/subspace") is false, /subspace reaches the commitment path whenever rs.ssStore == nil (SS disabled), and that path gets no concurrency limit — only the pair/byte caps. Either extend tryAcquireSubspaceQueryPermit to cover it, or state the SS-only scope in a comment on subspaceQuerySem so the gap reads as deliberate.
  • [suggestion] This is a fail-closed behavior change on a public query surface: a /store/{name}/subspace query that previously returned 5k pairs now returns ErrSubspaceCapExceeded instead of data. Worth an explicit release note so operators running indexers or tooling on wide prefixes know to narrow their prefixes or raise sc-subspace-max-pairs / sc-subspace-max-bytes before upgrading.
  • 3 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-cosmos/storev2/query/subspace.go Outdated
Comment thread sei-cosmos/storev2/query/subspace.go
Comment thread sei-db/config/toml.go
amir-deris and others added 2 commits August 26, 2026 14:48
Thread request context through IteratorOn on the SS path, centralize the
empty-prefix guard in ScanSubspace, and document <=0 default semantics
for subspace limits in app.toml.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris amir-deris changed the title Bound /store/*/subspace ABCI queries against memory-exhaustion DoS (PLT-799) Bound /store/*/subspace ABCI queries Aug 26, 2026
Comment thread sei-cosmos/storev2/query/subspace.go
amir-deris and others added 3 commits August 26, 2026 17:51
Cancelled or timed-out /subspace queries were panicking on the SS path
because state.Store fatally handles iterator-construction failures.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris
amir-deris enabled auto-merge August 26, 2026 16:46

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 260ccf0. Configure here.

if err != nil {
panic(fmt.Errorf("failed to marshal KV pairs: %w", err))
}
return bz, nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancelled subspace scans can succeed

High Severity

ScanSubspace exits the loop when Valid() becomes false and then marshals whatever pairs it already collected. It never checks iterator.Error() or ctx.Err() after iteration, so an SS MVCC skip that stops on deadline, cancel, or I/O is returned as a successful complete result. Clients get a code-0 truncated subspace instead of an error, which breaks the fail-closed contract.

Fix in Cursor Fix in Web

Triggered by learned rule: sei-db: iterator factories must return (dbm.Iterator, error), not swallow errors

Reviewed by Cursor Bugbot for commit 260ccf0. Configure here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth double clicking into this one @amir-deris

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants