feat(io): batched footer resolve on rest_ioctx - #188
Draft
ran-yuan-rui wants to merge 8 commits into
Draft
Conversation
Resolving N objects' parquet footers costs N independent blocking suffix probes, each on a fresh TCP+TLS connection (the synchronous metadata path shares DNS/TLS-session state but not live connections). Add rest_ioctx::resolve_footer_objects(paths, on_result, stop): batched submission with streamed per-entry completion. One curl multi driven on the caller's thread carries every probe (and HEAD fallback) of a batch, reusing its pooled connections across entries, with per-entry semantics identical to open_io_object(path, parquet_footer_probe): verified-206 window, 200/416/unverifiable-206 HEAD fallback, the same retry policy per entry, per-attempt re-authorization and ETag capture, and the same perf-snapshot attribution. Each result carries a stashless io_object (size + validation tag) plus the footer window as a separate payload whose buffer is a lease on an ioctx-wide byte budget: bytes return when the buffer is freed, bounding resolve-ahead memory without attaching budget-held state to long-lived objects. While any transfer is active the engine acquires budget non-blockingly; a blocking, stop-aware wait happens only at zero active transfers. Batches FIFO-serialize per ioctx; a queued batch cancels out of the queue without side effects. Delivery is exactly-once per input occurrence on the caller's thread; cancellation aborts in-flight transfers and delivers one operation_canceled per undelivered entry; a throwing callback cancels the remainder and rethrows the first exception after the sweep. Config: footer_resolve_max_inflight (default derives n_reactors * max_connections; 0 disables the API) and footer_resolve_stash_budget (default 2 * inflight * footer_probe_bytes). admission_control gains try_acquire plus reserved/peak accessors, and the ioctx perf snapshot reports the budget's live/peak bytes.
…gauges The connection-reuse case retains all twelve footer payloads in its results while sizing the stash budget at four windows; under the payload-is-the-lease contract the resolve call then blocks waiting for bytes only its own caller could free. Size the budget to the retained payload count. The budget case now also asserts the snapshot gauges directly (reserved == peak == budget while payloads are held; reserved drops to zero with peak still at budget after release), and the scheduling-loop checks use fixed assertion counts so the suite's case and assertion totals stay stable.
Memory now tracks the ledger: every fallback, terminal, and cancel path destroys the probe buffer before releasing its budget reservation, probe buffers reserve exactly the window up front so growth can never exceed the lease, and an explicit footer_resolve_stash_budget smaller than footer_probe_bytes is rejected at submission (a sub-window budget cannot be honored as a hard cap). Entry submission is exception-safe: an authorizer or curl setup failure becomes that entry's error and its siblings continue, and an unwind guard detaches any easy handle still attached to the batch multi before the owning entries are destroyed. Path parsing failures likewise become per-entry errors instead of aborting the batch with no callbacks. The event loop drops its full-vector scans for a backoff counter plus a deadline min-heap, giving O(N + R log N) scheduling instead of a worst-case O(N^2), and the per-submit clock read only happens when perf_instrumentation is on.
Three cases on the batched footer resolve: multiple malformed-206 objects under a two-window budget complete alongside healthy siblings with the reservation gauge back at zero; an authorizer that throws for one key delivers that exception to that entry alone, with no GET issued for it and every sibling succeeding; and an unparsable URI in the batch is isolated as that entry's error, while an explicit stash budget smaller than the probe window is rejected at submission. The loopback harness gains per-key malformed-206 scripting and the mock authorizer gains per-key exception injection.
After the first callback exception nothing may be delivered as success: completion draining, new-entry admission, and retry resubmission all stop at the first recorded throw (or a stop request), leaving undrained completions to the cancel sweep. A driver failure mid-batch (a curl multi error) now runs the same sweep before rethrowing, so every undelivered entry still receives exactly one canceled result on every exit path. An explicit footer_resolve_stash_budget of zero is no longer silently treated as the derived default — it falls under the sub-window rejection like any other too-small budget. And a zero footer_probe_bytes now matches the single-probe path exactly: batch entries skip the suffix GET and start at the HEAD fallback, with no lease taken.
…nobs A deterministic GET barrier on the loopback server holds three requests and releases them together, so a callback that throws on the first delivery faces simultaneously-ready siblings: the case pins one success, every remaining entry canceled, only the initial three GETs on the wire, the original exception rethrown, and no callback after return. An explicit zero stash budget throws before any authorization or network request, falling under sub-window rejection rather than reading as the derived default. And a zero probe window matches the single-probe path: no GET and no reservation per entry, one HEAD each, with size and quoted ETag from the HEAD and a null footer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Opening a parquet footer currently uses a one-shot blocking suffix GET. The process-wide CURLSH shares DNS and TLS-session state but deliberately does not share live connections, so the normal many-file path pays a separate TCP/TLS connection setup per file.
This adds
rest_ioctx::resolve_footer_objects(paths, on_result, stop). A per-call curl multi submits the probes concurrently, keeps at mostfooter_resolve_max_inflighttransfers active, and reuses idle connections within the call. Results are delivered one file at a time on the caller's thread; there is no whole-batch completion barrier. This PR adds the cuCascade mechanism only — wiring Sirius scan preparation to it follows after sirius-db/sirius#1340.Each entry follows the existing
open_io_object(path, parquet_footer_probe)behavior: verified 206 responses return the suffix window, unusable responses (200, 416, or an unverifiable 206) fall back to HEAD, a zero-byte window uses HEAD directly, and retries re-authorize each attempt while preserving ETag capture and perf-counter attribution.Once a batch is accepted, every input occurrence is delivered exactly once. Per-entry failures (4xx, credential errors, unparsable paths) do not cancel siblings. Cancellation reports
operation_canceledfor undelivered entries. If a callback throws, no later entry is reported as successful; the remaining entries are canceled before the first exception is rethrown. Submission validation errors (empty batch, the API disabled, a budget below one probe window) throw before any delivery.The returned
io_objectis stashless and carries only identity, size, and the validation tag. Footer bytes are returned separately under an ioctx-widefooter_resolve_stash_budgetlease, released when the payload buffer is freed. Concurrent calls are FIFO-serialized per ioctx, andfooter_resolve_max_inflight = 0disables the API.Supporting changes:
exec::admission_controlgains a non-blockingtry_acquireplusreserved()/peak_reserved()accessors, the ioctx perf snapshot reports the budget's live/peak bytes, and the loopback test server gains keep-alive, per-key request counters, per-key response scripting, and a deterministic GET barrier.Validation
The initial 14-case conformance suite was red against main; six cases added during review bring the final suite to 20. Coverage: byte/tag/window parity with the single-probe path including the HEAD fallback and the zero-window short-circuit, per-entry error isolation (4xx, authorizer exceptions, unparsable URIs), streamed completion order, exactly-once delivery including duplicates, cancellation before submission, mid-transfer, and during retry backoff, callback throws under simultaneously-ready completions, budget residency and gauges including sub-window rejection, single-worker and saturated consumers, FIFO fairness and queued-batch cancellation, lease-outlives-ioctx, connection reuse (12 GETs over at most 2 accepted connections in a no-retry keep-alive run), 503 retry, and snapshot parity with N single opens.
cucascade_io_tests: 135/135 green (1610 assertions).Sirius consumer check: cuCascade
c0a2607was consumed byintegrate_cucs_ioat8e7ce16b(sirius-db/sirius#1340). The full rebuild passed.make s3-testran 92 cases: 90 passed, while the same two retry-log observability cases tracked in #175 remained failing. This PR does not change those results.