Skip to content

enhancement(buffers): improve disk buffer lifecycle visibility - #26050

Open
Jansen-w wants to merge 3 commits into
vectordotdev:masterfrom
Jansen-w:fix/disk-buffer-orphan-safety
Open

enhancement(buffers): improve disk buffer lifecycle visibility#26050
Jansen-w wants to merge 3 commits into
vectordotdev:masterfrom
Jansen-w:fix/disk-buffer-orphan-safety

Conversation

@Jansen-w

@Jansen-w Jansen-w commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Report disk buffers under data_dir that are not referenced by the active configuration.
  • Run discovery on one dedicated worker outside Tokio, bounded to one active scan and one latest pending request, so filesystem traversal cannot delay startup, reload, or runtime shutdown.
  • Suppress stale scan results across rapid reloads and resolve symlinked roots, directories, and files without following cycles or paths outside the buffer root.
  • Schedule orphan diagnostics only after successful startup or reload. The immediate post-reload scan temporarily excludes paths from the previous authoritative configuration because removed sinks may still be draining them.
  • Log when existing disk buffers contain unread data and report approximate unread event/byte progress while removed sinks drain in the background.

Graceful shutdown behavior is intentionally unchanged here; that behavior is isolated in #26238.

Vector configuration

No standalone configuration is required. The behavior applies to sinks configured with disk buffers during startup and configuration reload.

How did you test this PR?

  • Disk-buffer encodability, filtering, overflow, and recovered-usage tests
  • Seven orphan-discovery tests covering configured paths, nested IDs, symlinked roots/files/directories, cycle avoidance, root escapes, stale-result suppression, request coalescing, and non-blocking scanner scheduling/drop
  • Disk-buffer topology reload regressions
  • Real disk-buffer drain test covering nonzero usage, acknowledgement, monitor completion, and lock release/reopen
  • Combined enhancement(buffers): improve disk buffer lifecycle visibility #26050/fix(topology): await removed disk sinks on shutdown #26238 topology suite: 65 passed, 1 ignored
  • cargo clippy -p vector-buffers --all-targets -- -D warnings
  • cargo clippy -p vector --lib --tests -- -D warnings
  • cargo fmt --all -- --check
  • Full GitHub Actions test, clippy, formatting, generated-doc, and changelog checks

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. A changelog fragment is included.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Related: #26238

@github-actions github-actions Bot added the domain: topology Anything related to Vector's topology code label Aug 7, 2026
@Jansen-w Jansen-w closed this Aug 10, 2026
@Jansen-w
Jansen-w deleted the fix/disk-buffer-orphan-safety branch August 10, 2026 19:42
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 10, 2026
@Jansen-w
Jansen-w restored the fix/disk-buffer-orphan-safety branch August 10, 2026 19:43
@Jansen-w Jansen-w reopened this Aug 10, 2026
@Jansen-w
Jansen-w force-pushed the fix/disk-buffer-orphan-safety branch 4 times, most recently from a6a2bc4 to b7df047 Compare August 12, 2026 16:58
@Jansen-w
Jansen-w force-pushed the fix/disk-buffer-orphan-safety branch from fd18659 to 7b9f6e0 Compare August 27, 2026 22:02
@Jansen-w
Jansen-w marked this pull request as ready for review August 27, 2026 22:22
@Jansen-w
Jansen-w requested a review from a team as a code owner August 27, 2026 22:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: topology Anything related to Vector's topology code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant