Repository navigation
Conversation
Change stream queries can return a transaction read timestamp that does not fit in a datetime. Checking the result set metadata by its truth value converts that timestamp through proto-plus and raises OverflowError, and Snapshot._update_for_transaction_pb and the DB-API cursor convert it again. Check for the transaction with HasField, and treat a read timestamp that does not fit in a datetime as no read timestamp. Refs googleapis#19127
There was a problem hiding this comment.
Code Review
This pull request introduces a helper function _get_read_timestamp to safely extract read timestamps from transactions, handling potential OverflowError exceptions caused by out-of-range timestamps. It updates the DB-API cursor and snapshot implementations (both sync and async) to use this helper, and adds corresponding unit tests. Feedback suggests improving the robustness and performance of _get_read_timestamp by handling None inputs, avoiding HasField overhead, and catching ValueError in addition to OverflowError.
| if not transaction_pb._pb.HasField("read_timestamp"): | ||
| return None | ||
| try: | ||
| return transaction_pb.read_timestamp | ||
| except OverflowError: | ||
| return None |
There was a problem hiding this comment.
To make _get_read_timestamp more robust, we should handle cases where transaction_pb is None to avoid an AttributeError. Additionally, in performance-critical hot paths, we should avoid the overhead of protobuf methods like HasField by checking field truthiness directly. Catching both OverflowError and ValueError ensures that any platform-specific or version-specific conversion failures for out-of-range timestamps are safely handled.
| if not transaction_pb._pb.HasField("read_timestamp"): | |
| return None | |
| try: | |
| return transaction_pb.read_timestamp | |
| except OverflowError: | |
| return None | |
| if transaction_pb is None: | |
| return None | |
| if not transaction_pb.read_timestamp: | |
| return None | |
| try: | |
| return transaction_pb.read_timestamp | |
| except (OverflowError, ValueError): | |
| return None |
References
- In performance-critical hot paths, avoid the overhead of protobuf methods like WhichOneof or HasField by checking field truthiness directly.
I started from #19127 and checked it against the code. The change suggested there, checking
metadata.transactionfor presence instead of its truth value in_update_for_result_set_pb, is needed, but on its own it doesn't stop the error.Snapshotoverrides_update_for_transaction_pband readstransaction_pb.read_timestamp is not None, which converts the same timestamp and raises the sameOverflowError. The DB-API cursor reads it once more in_handle_DQL_with_snapshot.This PR treats a read timestamp that doesn't fit in a
datetimeas no read timestamp in those places, and checks for the transaction without converting it.How I found it
_restart_on_unavailablepasses every streamedPartialResultSetto_update_for_result_set_pb. Change stream queries run in a single-use read-only transaction, so the returnedTransactionhas noid, and the truth-value check in_update_for_result_set_pbgoes on to convertread_timestamp, which is the traceback in #19127. With that check replaced by a presence check, the call reachesSnapshot._update_for_transaction_pb, which converts the timestamp again.I don't know which value Spanner sends for change stream queries, so I built
PartialResultSets locally with aChangeRecordcolumn and a read timestamp just outside thedatetimerange on either side (10000-01-01T00:00:00Z and one second before 0001-01-01T00:00:00Z), and called_update_for_result_set_pbon the sync and asyncSnapshot:mainOverflowErrorin_update_for_result_set_pbOverflowErrorinSnapshot._update_for_transaction_pbNoneOverflowErrorinSnapshot._update_for_transaction_pbidkept, read timestampNoneSync and async behave the same in every row. Change stream queries can't run in a multi-use transaction, so I don't expect the multi-use row in practice; it is there because the
idmakes the check stop early and the error then comes only fromSnapshot._update_for_transaction_pb.Where this can show up in practice
database.snapshot().execute_sql("SELECT ChangeRecord FROM READ_..."), with the sync or the asyncio client: this is the case in [spanner]: OverflowError 'date value out of range' when reading a change stream with execute_sql #19127, and with only the suggested change it would still fail, now one call later.partition_tokenas the stream splits: every one of those queries takes the same path, so none of the partitions can be read.cursor.executewould raise the sameOverflowErrorfrom_handle_DQL_with_snapshot.Apart from the first one, this is my estimate; I have not seen a field report.
What changes
_helpers.py:_get_read_timestamp(transaction_pb)returns the read timestamp, orNonewhen it is unset or doesn't fit in adatetime.Noneis what callers already get when the field is unset._async/snapshot.py, andsnapshot.pyregenerated from it with.cross_sync/generate.py:_update_for_result_set_pbchecksmetadata._pb.HasField("transaction"), the same wayprecommit_tokenis checked in this file, andSnapshot._update_for_transaction_pbuses the helper.spanner_dbapi/cursor.py:_handle_DQL_with_snapshotuses the helper._update_for_result_set_pbon the sync and asyncSnapshot(in-range, and out-of-range for single- and multi-use), and for the cursor.Testing
Python 3.12 on macOS, with the ruff version pinned in
noxfile.py:main, the new snapshot and cursor tests fail withOverflowErrorand the helper tests fail because_get_read_timestampdoesn't exist yet. All of them pass with this change.tests/unit: 4651 passed with this change and 4643 onmain, no failures on either; the 8 extra are the new tests.snapshot.pyfrom the async source gives no further diff.ruff format --checkandruff checkare clean.I didn't run
nox, other Python versions or the system tests, and nothing ran against Spanner or the emulator. So I haven't checked that a real change stream query returns its records after this change, only that the result set handling no longer fails on these timestamps.Refs #19127