fix(buffers): publish disk buffer progress before notifying readers

[graphcareful]

Summary

Ensure disk buffer readers are notified only after flushed writer progress has been published to the shared ledger.

This consolidates record ID, buffer size, and usage accounting with reader notification for both explicit and implicit flushes. It prevents readers from consuming and acknowledging newly readable bytes while the ledger still contains stale writer state, which could corrupt accounting and stall buffer progress.

Vector configuration

No new configuration is introduced. The change applies to existing disk-buffered components, for example:

sinks:
  out:
    buffer:
      type: disk

How did you test this PR?

Using new antithesis runs located in the branch https://github.com/graphcareful/vector/tree/test/disk-buffer-oversized-progress two assertions were firing:

• https://github.com/graphcareful/vector/blob/test/disk-buffer-oversized-progress/tests/antithesis/harness/src/bin/eventually_disk_progress_check.rs#L304-L308
• https://github.com/graphcareful/vector/blob/test/disk-buffer-oversized-progress/tests/antithesis/harness/src/bin/eventually_disk_progress_check.rs#L290-L294

After discovering the root cause had nothing to do with oversized records, it was tracked down to the final batch remaining stuck in the disk buffer.

With this change the two antithesis failures no longer fire.

• cargo test -p vector-buffers --no-default-features (77 passed, 2 pre-existing ignored)
• cargo test -p vector-buffers publishing_writer_progress_updates_ledger_before_waking_reader --no-default-features

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. A changelog fragment is included.
• No. A maintainer will apply the no-changelog label to this PR.

References

None.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdbbd6e0f9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[graphcareful]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: c390fab7c7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".