Skip to content

fix: account polling elapsed time on the monotonic clock - #204

Open
yzxcj797 wants to merge 1 commit into
xai-org:mainfrom
yzxcj797:fix/poll-timer-monotonic-203
Open

fix: account polling elapsed time on the monotonic clock#204
yzxcj797 wants to merge 1 commit into
xai-org:mainfrom
yzxcj797:fix/poll-timer-monotonic-203

Conversation

@yzxcj797

Copy link
Copy Markdown

Fixes #203.

Summary

PollTimer measured elapsed duration with time.time() — a wall clock. NTP corrections, manual clock changes or VM suspend/restore move it:

  • a forward adjustment can make an otherwise healthy deferred chat / collection indexing / video-generation poll time out immediately (the issue's repro: a 9900s jump yields TimeoutError: Polling timed out after 9900.0s on a poll that had run for ~0s);
  • a backward adjustment silently extends polling beyond the caller's requested timeout.

Both the sync and async polling paths share this helper. The fix uses time.monotonic() — which cannot go backward — for the start timestamp and the elapsed-time computation. Internal change only: no public API and no normal-case timing difference.

Testing

New tests/poll_timer_test.py (deterministic, mocks only the clock):

  • forward jump — the issue's exact shape (10s timeout, 20s interval, wall clock leaps between construction and the first sleep computation): returns 9.5 instead of raising;
  • backward jump with real elapsed time — 15s actually elapse while the wall clock moves back: the poll still times out instead of quietly polling forever;
  • timeout on genuine elapsed time — pins that real timeouts still raise with the correct message.

3/3 pass with the fix; differential with poll_timer.py reverted fails all three (the first returns the full interval, the others never raise).

PollTimer measured elapsed duration with time.time(), a wall clock.
NTP corrections, manual clock changes or VM suspend/restore move it:
a forward adjustment can make an otherwise healthy deferred chat,
collection indexing or video-generation poll time out immediately, and
a backward adjustment can extend polling beyond the caller's requested
timeout. Both the sync and async polling paths share this helper.

Use time.monotonic(), which cannot go backward, for the start
timestamp and the elapsed-time computation. No public API change and
no normal-case timing change.

Fixes xai-org#203
@yzxcj797
yzxcj797 requested a review from a team as a code owner August 22, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Polling timeouts can misfire after wall-clock adjustments

1 participant