Skip to content

fix(spanner): handle out-of-range read timestamps in result sets - #19131

Open
0z5a wants to merge 1 commit into
googleapis:mainfrom
0z5a:spanner-read-timestamp-out-of-range
Open

0z5a wants to merge 1 commit into
googleapis:mainfrom
0z5a:spanner-read-timestamp-out-of-range

Conversation

@0z5a

@0z5a 0z5a commented Nov 19, 2026

Copy link
Copy Markdown

I started from #19127 and checked it against the code. The change suggested there, checking metadata.transaction for presence instead of its truth value in _update_for_result_set_pb, is needed, but on its own it doesn't stop the error. Snapshot overrides _update_for_transaction_pb and reads transaction_pb.read_timestamp is not None, which converts the same timestamp and raises the same OverflowError. The DB-API cursor reads it once more in _handle_DQL_with_snapshot.

This PR treats a read timestamp that doesn't fit in a datetime as no read timestamp in those places, and checks for the transaction without converting it.

How I found it

_restart_on_unavailable passes every streamed PartialResultSet to _update_for_result_set_pb. Change stream queries run in a single-use read-only transaction, so the returned Transaction has no id, and the truth-value check in _update_for_result_set_pb goes on to convert read_timestamp, which is the traceback in #19127. With that check replaced by a presence check, the call reaches Snapshot._update_for_transaction_pb, which converts the timestamp again.

I don't know which value Spanner sends for change stream queries, so I built PartialResultSets locally with a ChangeRecord column and a read timestamp just outside the datetime range on either side (10000-01-01T00:00:00Z and one second before 0001-01-01T00:00:00Z), and called _update_for_result_set_pb on the sync and async Snapshot:

case main the #19127 change only this PR
single-use, out of range OverflowError in _update_for_result_set_pb OverflowError in Snapshot._update_for_transaction_pb read timestamp None
multi-use, out of range OverflowError in Snapshot._update_for_transaction_pb same id kept, read timestamp None
in-range timestamp read timestamp set read timestamp set read timestamp set

Sync and async behave the same in every row. Change stream queries can't run in a multi-use transaction, so I don't expect the multi-use row in practice; it is there because the id makes the check stop early and the error then comes only from Snapshot._update_for_transaction_pb.

Where this can show up in practice

  • Reading a change stream with database.snapshot().execute_sql("SELECT ChangeRecord FROM READ_..."), with the sync or the asyncio client: this is the case in [spanner]: OverflowError 'date value out of range' when reading a change stream with execute_sql #19127, and with only the suggested change it would still fail, now one call later.
  • A service that reads change streams itself instead of through Dataflow, querying each child partition by its partition_token as the stream splits: every one of those queries takes the same path, so none of the partitions can be read.
  • A change stream query through the DB-API in autocommit mode, where the cursor runs it in a single-use snapshot: once the snapshot no longer fails, cursor.execute would raise the same OverflowError from _handle_DQL_with_snapshot.

Apart from the first one, this is my estimate; I have not seen a field report.

What changes

  • _helpers.py: _get_read_timestamp(transaction_pb) returns the read timestamp, or None when it is unset or doesn't fit in a datetime. None is what callers already get when the field is unset.
  • _async/snapshot.py, and snapshot.py regenerated from it with .cross_sync/generate.py: _update_for_result_set_pb checks metadata._pb.HasField("transaction"), the same way precommit_token is checked in this file, and Snapshot._update_for_transaction_pb uses the helper.
  • spanner_dbapi/cursor.py: _handle_DQL_with_snapshot uses the helper.
  • Tests for the helper, for _update_for_result_set_pb on the sync and async Snapshot (in-range, and out-of-range for single- and multi-use), and for the cursor.

Testing

Python 3.12 on macOS, with the ruff version pinned in noxfile.py:

  • On main, the new snapshot and cursor tests fail with OverflowError and the helper tests fail because _get_read_timestamp doesn't exist yet. All of them pass with this change.
  • tests/unit: 4651 passed with this change and 4643 on main, no failures on either; the 8 extra are the new tests.
  • Regenerating snapshot.py from the async source gives no further diff. ruff format --check and ruff check are clean.

I didn't run nox, other Python versions or the system tests, and nothing ran against Spanner or the emulator. So I haven't checked that a real change stream query returns its records after this change, only that the result set handling no longer fails on these timestamps.

Refs #19127

Change stream queries can return a transaction read timestamp that does
not fit in a datetime. Checking the result set metadata by its truth
value converts that timestamp through proto-plus and raises
OverflowError, and Snapshot._update_for_transaction_pb and the DB-API
cursor convert it again.

Check for the transaction with HasField, and treat a read timestamp
that does not fit in a datetime as no read timestamp.

Refs googleapis#19127
@0z5a
0z5a requested a review from a team as a code owner November 19, 2026 04:42

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a helper function _get_read_timestamp to safely extract read timestamps from transactions, handling potential OverflowError exceptions caused by out-of-range timestamps. It updates the DB-API cursor and snapshot implementations (both sync and async) to use this helper, and adds corresponding unit tests. Feedback suggests improving the robustness and performance of _get_read_timestamp by handling None inputs, avoiding HasField overhead, and catching ValueError in addition to OverflowError.

Comment on lines +834 to +839
if not transaction_pb._pb.HasField("read_timestamp"):
return None
try:
return transaction_pb.read_timestamp
except OverflowError:
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make _get_read_timestamp more robust, we should handle cases where transaction_pb is None to avoid an AttributeError. Additionally, in performance-critical hot paths, we should avoid the overhead of protobuf methods like HasField by checking field truthiness directly. Catching both OverflowError and ValueError ensures that any platform-specific or version-specific conversion failures for out-of-range timestamps are safely handled.

Suggested change
if not transaction_pb._pb.HasField("read_timestamp"):
return None
try:
return transaction_pb.read_timestamp
except OverflowError:
return None
if transaction_pb is None:
return None
if not transaction_pb.read_timestamp:
return None
try:
return transaction_pb.read_timestamp
except (OverflowError, ValueError):
return None
References
  1. In performance-critical hot paths, avoid the overhead of protobuf methods like WhichOneof or HasField by checking field truthiness directly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant