Skip to content

Bound object-store census for cell diagnostics - #918

Draft
zlareb1 wants to merge 2 commits into
feat/cell-bounded-process-censusfrom
feat/cell-bounded-bucket-probes
Draft

zlareb1 wants to merge 2 commits into
feat/cell-bounded-process-censusfrom
feat/cell-bounded-bucket-probes

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Related: #915

Changelog category (leave one):

  • Improvement

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Internal cell diagnostics exhaust bounded object-store prefixes, bound responses before parsing and delete an owned marker only with its exact contents and entity tag.

Why

A complete cell-separation check needs bounded listing pages and a marker cleanup that refuses a concurrent replacement. Ordinary object reads materialize response bodies and ordinary deletes do not provide that proof.

Scope and risk

Adds fixed-purpose listing and own-marker primitives through the existing cached client, shared executor and listing or transfer leases. A diagnostic client variant permits one request attempt, checks the installed SDK source and service model, bounds raw responses before the existing parser, and requires the store's response date. Its four operation hooks refuse implicit region redirects and unsupported signing before they can start another operation. Ordinary clients keep their current configuration and handlers.

Listing responses are limited to 8 MiB, operation errors to 64 KiB and marker reads to 1025 bytes. Marker contents are limited to 1024 bytes. Cleanup reads contents and the strong entity tag together, then uses one conditional delete; an unknown result or replacement remains incomplete. No arbitrary peer bucket, prefix, key or operation is accepted.

This component has no wire operation or activation caller. The configured-root walk inspects every permitted page and refuses a foreign cell marker at any depth. It accepts at most 100,000 objects, 100 pages and 16 MiB of UTF-8 key bytes per invocation; an incomplete final page at a limit refuses without another fetch. Authenticated peer barriers, shared-wait integration and actual store enforcement remain required before a physical-separation result can be claimed. The SDK identity check deliberately refuses unsupported installed versions or models.

Compatibility, rollout and rollback

  • Wire: no operation, field or error changes.
  • Operator steps: none for this internal component. No setting, migration or stored content changes.
  • Rollback: removes the diagnostic primitives and cache discriminator; ordinary clients retain their existing behavior.

Tests

  • New diagnostic and complete-prefix contracts plus six existing client and lease controls: 91 passed. Actual SDK parser, event ordering and signed-request transport seams use substituted transport and synthetic credentials; no network or store request is made.
  • One malformed hand-built HTTP response in a new event test initially failed before the guard. Correcting its response shape preserves all assertions; the same 68 cases pass.
  • Scoped Ruff lint, formatting, statement-count checks and strict mypy over both source files pass. The complete-prefix checks cover exact object, page and byte limits, overflow, foreign markers at any depth, incomplete pages and original cancellation identity. All 91 cases pass with source and Git state unchanged.
  • Independent source reviews of the primitives and complete-prefix consumer have no findings or nits. A fresh review of the combined pushed head and full CI remain pending.
  • Real store conditional-delete enforcement, complete listing consistency, delayed remote completion, native attachment and mixed-version behavior remain unrun. This PR makes no physical-separation or complete attachment claim.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (Codex): source, SDK path inspection, independent review, bounded transport and cleanup contracts, and scoped checks.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Small related contracts and scoped static checks pass on the pushed source.
  • Commits are signed off.
  • Full GitHub CI and final pushed-head review pass before merge.

Use the shared client and executor for fixed diagnostic operations. Refuse oversized or unknown responses before parsing and protect marker cleanup with the exact contents and entity tag.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Read every permitted page before accepting marker absence. Bound the complete walk by objects, pages and UTF-8 key bytes, and refuse a foreign cell marker at any depth.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1 zlareb1 changed the title Bound object-store responses for cell diagnostics Bound object-store census for cell diagnostics Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant