DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension#770
DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension#770nikagra wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Implements negotiation and support for Scylla’s SCYLLA_USE_METADATA_ID protocol extension to enable metadata-id based skip_meta behavior (backporting CQL v5 prepared-statement metadata-id semantics to earlier protocol versions).
Changes:
- Adds
SCYLLA_USE_METADATA_IDparsing fromSUPPORTEDand includes it inSTARTUPwhen negotiated. - Extends protocol encode/decode to read/write
result_metadata_idfor PREPARE/EXECUTE on pre-v5 when the extension is used, and fixes on-wire encoding of_SKIP_METADATA_FLAG. - Updates execution/result handling to conditionally use
skip_metaand to refresh cached prepared metadata when the server reports metadata changes.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
cassandra/protocol_features.py |
Adds the SCYLLA_USE_METADATA_ID feature flag and includes it in negotiated STARTUP options. |
cassandra/protocol.py |
Writes _SKIP_METADATA_FLAG in query params; adds pre-v5 extension handling for result_metadata_id in PREPARE/EXECUTE. |
cassandra/cluster.py |
Adjusts when skip_meta is enabled and updates cached prepared metadata/id on METADATA_CHANGED responses. |
tests/unit/test_protocol_features.py |
Adds unit tests for feature parsing and STARTUP option inclusion. |
tests/unit/test_protocol.py |
Adds unit tests for skip-meta flag encoding and metadata-id handling in pre-v5 PREPARE/EXECUTE paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| elif self.result_metadata_id is not None: | ||
| write_string(f, self.result_metadata_id) |
There was a problem hiding this comment.
ExecuteMessage.send_body() now writes result_metadata_id for protocol versions that don’t use standard prepared-metadata (pre-v5) whenever self.result_metadata_id is non-None. For pre-v5 this extra field is only valid when SCYLLA_USE_METADATA_ID was negotiated on that connection; otherwise it changes the wire layout (the server will interpret the metadata id bytes as the query parameters) and the request will fail.
To make this safe, ensure the decision to include result_metadata_id is gated by the negotiated feature (e.g., add an explicit use_metadata_id/send_result_metadata_id flag on the message that the caller sets based on connection.features.use_metadata_id, or guarantee centrally that result_metadata_id is cleared unless the extension is active for that connection).
| elif self.result_metadata_id is not None: | |
| write_string(f, self.result_metadata_id) |
There was a problem hiding this comment.
Agreed. The elif self.result_metadata_id is not None path in send_body is now only reached when the caller explicitly set the field — which only happens in _query() after confirming connection.features.use_metadata_id (or CQL v5). For any connection that didn't negotiate the extension, result_metadata_id remains None and the branch is never taken, so the wire layout is unaffected.
There was a problem hiding this comment.
There is still a problem here. If the result id feature is negotiated on the connection, then you need to ALWAYS send some result metadata id in EXECUTE. Skipping the write_string will result in a protocol error.
Your use_metadata_id may be False even if extension was negotiated, if the server decided to skip the metadata in PREPARED response. In such case, you'll skip writing the id here, and encounter protocol error.
Even if you fix this specific case, there is still possibility of mixed cluster, with some nodes supporting the extension. In that case result_metadata_id will be None, and if you send to a node that has the extension negotiated, you'll again not send the id and encounter protocol error.
To sum up: this serialization here should check if feature is negotiated, and base sending this field only on that.
There was a problem hiding this comment.
👨💻: Addressed your concerns
🤖:
Automated response
Fixed. Added a `use_metadata_id` boolean flag to `ExecuteMessage` (default `False`). `send_body` now gates the `result_metadata_id` field on `ProtocolVersion.uses_prepared_metadata(protocol_version) or self.use_metadata_id` — both branches unified — writing an empty sentinel (`b''`) when the hash is `None` rather than omitting the field or crashing.
In _query(), message.use_metadata_id = connection.features.use_metadata_id is set unconditionally after borrowing the connection, so the decision is always based on what was negotiated on that specific connection.
This also fixes a second issue: the old separate v5 branch would crash with TypeError when result_metadata_id is None (LWT statement on v5, or statement prepared before the extension was active in a mixed cluster). The unified sentinel handles that case correctly on both code paths.
Two new tests cover this: test_execute_message_scylla_metadata_id_none_writes_sentinel (extension active, None hash → sentinel on v4) and test_execute_message_v5_metadata_id_none_writes_sentinel (v5 native path, None hash → sentinel instead of crash).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
ade35d8 to
f42e225
Compare
|
I'm not sure where, but we should document this - with reference mainly to the scylladb docs about this feature. |
|
@mykaul Documentation I'm aware of is MetadataId extension in CQLv4 Requirement Document |
dkropachev
left a comment
dkropachev
left a comment
There was a problem hiding this comment.
One blocking correctness issue below: skip_meta is being enabled for prepared statements that can still have empty/absent cached result metadata.
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
6eea397 to
a86fd53
Compare
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
7ba5835 to
a86fd53
Compare
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
a86fd53 to
8880f03
Compare
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
170fd31 to
5fe1902
Compare
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
fcd3eba to
5fe1902
Compare
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
5fe1902 to
251b1a8
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughThis PR adds Scylla's SCYLLA_USE_METADATA_ID support: negotiates a new protocol feature, encodes/decodes result_metadata_id and skip_meta/use_metadata_id in QUERY/EXECUTE/PREPARE/RESULT messages, defers attaching metadata-id info until a connection is borrowed, and updates prepared-statement cached metadata when servers send new result_metadata_id. It includes docs and unit tests covering v4/v5 and the Scylla extension paths. Sequence DiagramsequenceDiagram
participant Client
participant Connection
participant Server
Client->>Connection: execute(prepared_statement)
activate Connection
Connection->>Connection: ResponseFuture._query(): check cached metadata<br/>& connection features
alt has cached metadata & supports metadata-id
Connection->>Server: ExecuteMessage(skip_meta=True, result_metadata_id=cached_id)
else no cached metadata or unsupported
Connection->>Server: ExecuteMessage(skip_meta=False, result_metadata_id=None)
end
activate Server
alt metadata unchanged
Server->>Connection: ResultMessage(result_metadata_id=cached_id, no column_metadata)
else metadata changed
Server->>Connection: ResultMessage(result_metadata_id=new_id, column_metadata=[...])
end
deactivate Server
Connection->>Connection: ResponseFuture._set_result(): if result_metadata_id present,<br/>update prepared_statement metadata cache
deactivate Connection
Connection->>Client: return rows with cached metadata
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Actionable comments posted: 0 |
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
2 times, most recently
from
de8d3fc to
bfc9760
Compare
|
🤖: All review issues (1–9) have been addressed and the branch has been squashed to the required 2-commit shape. Requesting re-review from @Lorak-mmk and @dkropachev. What changed since last review: Production commit (
Test commit (
CI on the 8-commit pre-squash branch: 18/19 passed; |
Backport the prepared-statement metadata-ID mechanism from CQL v5 to earlier protocol versions via the SCYLLA_USE_METADATA_ID Scylla extension. protocol_features.py: - Add USE_METADATA_ID constant and use_metadata_id field to ProtocolFeatures - parse_from_supported: detect SCYLLA_USE_METADATA_ID in SUPPORTED options - add_startup_options: echo the extension back in STARTUP when negotiated protocol.py: - _write_query_params: fix _SKIP_METADATA_FLAG — it was stored but never written on the wire (dead code); now correctly sets the flag in the frame - recv_results_prepared: read result_metadata_id when the Scylla extension is active (pre-v5), in addition to the existing CQL v5 path - ExecuteMessage: add use_metadata_id flag (default False); send_body gates the result_metadata_id field on uses_prepared_metadata(protocol_version) OR use_metadata_id, writing an empty sentinel (b'') when the hash is unavailable (LWT / mixed cluster) instead of omitting the field entirely or crashing with TypeError cluster.py: - _create_response_future: build ExecuteMessage with defaults (skip_meta=False, result_metadata_id=None); set these in _query() once the connection is known - _query: after borrowing the connection set can_skip_meta, skip_meta, result_metadata_id, and use_metadata_id on the ExecuteMessage based on the actual connection's negotiated features; skip_meta is only enabled when the prepared statement has both a result_metadata_id AND cached result_metadata (guards against LWT/NO_METADATA statements and mixed-cluster re-prepare) - _set_result: on METADATA_CHANGED (new result_metadata_id in EXECUTE response) update prepared_statement.result_metadata then result_metadata_id in that order — safe write ordering so a concurrent reader using the old id gets full metadata from the server while a reader seeing the new id has the correct cached metadata immediately docs/scylla-specific.rst: document the extension and its behaviour
test_protocol_features.py: - test_use_metadata_id_parsing: SCYLLA_USE_METADATA_ID parsed from SUPPORTED - test_use_metadata_id_missing: use_metadata_id is False when key is absent - test_use_metadata_id_startup_options: key present in STARTUP when negotiated - test_use_metadata_id_not_in_startup_when_not_negotiated: absent otherwise test_protocol.py -- ExecuteMessage wire encoding: - test_execute_message_skip_meta_flag: _SKIP_METADATA_FLAG (0x02) written on v4 - test_execute_message_v5_skip_meta_sets_flag: _SKIP_METADATA_FLAG written in the 4-byte v5 flags word; confirms the flag was dead code in upstream before this PR - test_execute_message_scylla_metadata_id_v4: result_metadata_id written on v4 when use_metadata_id=True (Scylla extension) - test_execute_message_scylla_metadata_id_none_writes_sentinel: extension active but result_metadata_id=None writes empty sentinel b'' (LWT / mixed cluster) - test_execute_message_v5_metadata_id_none_writes_sentinel: v5 with result_metadata_id=None writes empty sentinel instead of TypeError crash test_protocol.py -- ResultMessage decoding: - test_recv_results_prepared_scylla_extension_reads_metadata_id: result_metadata_id read from PREPARE response on v4 when extension is active - test_recv_results_prepared_no_extension_skips_metadata_id: result_metadata_id not read on v4 without extension - test_recv_results_prepared_v5_reads_metadata_id: result_metadata_id read on v5 via the native uses_prepared_metadata() path (use_metadata_id=False) - test_recv_results_metadata_changed_flag: _METADATA_ID_FLAG in ROWS response causes result_metadata_id to be read and stored - test_recv_results_metadata_no_metadata_flag_skips_metadata_id: NO_METADATA early return leaves result_metadata_id unset (asserts not hasattr, not merely None) test_response_future.py -- _set_result METADATA_CHANGED update path: - test_set_result_updates_metadata_when_metadata_changed: both result_metadata and result_metadata_id updated when server sends a new metadata id - test_set_result_does_not_update_metadata_when_metadata_id_absent: cached metadata untouched when response carries no new metadata id (normal skip-meta path) - test_set_result_warns_when_metadata_id_but_no_column_metadata: warning emitted and result_metadata_id updated when new id arrives with empty column list - test_set_result_warns_when_metadata_id_but_column_metadata_is_none: same as above but with column_metadata=None (absent) rather than [] test_response_future.py -- _execute_after_prepare reprepare path: - test_repeat_orig_query_after_succesful_reprepare: fixed result_metadata_id type str->bytes; added assertion that value is stored on prepared_statement - test_execute_after_prepare_updates_result_metadata_id: both result_metadata and result_metadata_id refreshed from reprepare response when extension is active - test_execute_after_prepare_no_metadata_id_in_response: result_metadata_id left unchanged when reprepare response carries no metadata id test_response_future.py -- _query per-connection feature gating (6 scenarios): - test_query_sets_skip_meta_with_scylla_extension: skip_meta=True when connection has negotiated SCYLLA_USE_METADATA_ID and statement has cached metadata - test_query_no_skip_meta_without_extension: skip_meta=False when use_metadata_id is False; fixture uses truthy result_metadata to isolate this guard correctly - test_query_no_skip_meta_when_prepared_statement_has_no_metadata_id: skip_meta False when result_metadata_id=None (prepared before extension was active) - test_query_sets_skip_meta_for_protocol_v5: skip_meta=True on v5 via the native uses_prepared_metadata() path (use_metadata_id=False) - test_query_no_skip_meta_when_result_metadata_is_none: LWT guard -- skip_meta False when result_metadata=None despite extension being active
nikagra
force-pushed
the
driver-153-scylla-use-metadata-id
branch
from
bfc9760 to
d3300e2
Compare
| class ExecuteMessage(_QueryMessage): | ||
| opcode = 0x0A | ||
| name = 'EXECUTE' | ||
|
|
||
| def __init__(self, query_id, query_params, consistency_level, | ||
| serial_consistency_level=None, fetch_size=None, | ||
| paging_state=None, timestamp=None, skip_meta=False, | ||
| continuous_paging_options=None, result_metadata_id=None): | ||
| continuous_paging_options=None, result_metadata_id=None, | ||
| use_metadata_id=False): |
There was a problem hiding this comment.
Here ExecuteMessage got a new parameter, use_metadata_id, in addition to existing result_metadata_id
| elif isinstance(query, BoundStatement): | ||
| prepared_statement = query.prepared_statement | ||
| message = ExecuteMessage( | ||
| prepared_statement.query_id, query.values, cl, | ||
| serial_cl, fetch_size, paging_state, timestamp, | ||
| skip_meta=bool(prepared_statement.result_metadata), | ||
| continuous_paging_options=continuous_paging_options, | ||
| result_metadata_id=prepared_statement.result_metadata_id) | ||
| continuous_paging_options=continuous_paging_options) | ||
| elif isinstance(query, BatchStatement): |
There was a problem hiding this comment.
But here you remove result_metadata_id and skip_meta from this call to ExecuteMessage. Why?
| if self.prepared_statement and isinstance(message, ExecuteMessage): | ||
| has_result_metadata_id = self.prepared_statement.result_metadata_id is not None | ||
| has_result_metadata = bool(self.prepared_statement.result_metadata) | ||
| can_skip_meta = has_result_metadata_id and has_result_metadata and ( | ||
| ProtocolVersion.uses_prepared_metadata(connection.protocol_version) | ||
| or connection.features.use_metadata_id | ||
| ) | ||
| message.skip_meta = can_skip_meta | ||
| message.result_metadata_id = self.prepared_statement.result_metadata_id if can_skip_meta else None | ||
| message.use_metadata_id = connection.features.use_metadata_id | ||
|
|
There was a problem hiding this comment.
Ah, so instead of setting those when creating ExecuteMessage you started to set them here.
This seems a bit inelegant. Why did you decide on this approach? Can't we still pass those params to the constructor? And if we are going with your approach, why do we need those params in the constructor?
| def send_body(self, f, protocol_version): | ||
| write_string(f, self.query_id) | ||
| if ProtocolVersion.uses_prepared_metadata(protocol_version): | ||
| write_string(f, self.result_metadata_id) | ||
| if ProtocolVersion.uses_prepared_metadata(protocol_version) or self.use_metadata_id: | ||
| write_string(f, self.result_metadata_id if self.result_metadata_id is not None else b'') | ||
| self._write_query_params(f, protocol_version) |
There was a problem hiding this comment.
A more elegant approach, it seems to me, is to not have use_metadata_id in ExecuteMessage. Instead, pass ProtocolFeatures to this method, use it and version to decide wheter to send an id, and use it from ExecuteMessage
| new_result_metadata_id = getattr(response, 'result_metadata_id', None) | ||
| if self.prepared_statement and new_result_metadata_id is not None: | ||
| if response.column_metadata: | ||
| # Write result_metadata before result_metadata_id intentionally: | ||
| # a concurrent reader that still sees the old metadata_id will | ||
| # ask the server for full metadata and recover safely; a reader | ||
| # that sees the new metadata_id together with the new metadata | ||
| # is immediately correct. The opposite write order could expose | ||
| # a window where a reader uses a new metadata_id with stale metadata. | ||
| # Note: correctness of this ordering relies on CPython's GIL making | ||
| # individual attribute reads/writes effectively atomic. Other Python | ||
| # implementations (PyPy, Jython, etc.) may not provide this guarantee. | ||
| self.prepared_statement.result_metadata = response.column_metadata | ||
| else: | ||
| log.warning( | ||
| "Server sent a new result_metadata_id but no column metadata " | ||
| "for prepared statement %r. The cached column metadata will not " | ||
| "be updated; only result_metadata_id is refreshed.", | ||
| getattr(self.prepared_statement, 'query_id', None) | ||
| ) | ||
| self.prepared_statement.result_metadata_id = new_result_metadata_id |
There was a problem hiding this comment.
What is the concurrency model here? I don't think this code works if its possible for it to execute concurrently.
Consider the following:
- We start with schema X and perform two schema changes, to version Y and then Z.
- We issue two requests
- First request receives
Y_idandY, setsresult_metadatatoY, and then execution switches to second request - Second request receives
Z_idandZ, setsresult_metadatatoZ, thenresult_metadata_idtoZ_id. - Execution switches back to first request, it sets
result_metadata_idtoY_id.
Now we have result_metadata = Z, and result_metadata_id - Y_id. Any request executed won't receive new metadata (because id is the newest) but will try to deserialize results using older metadata.
| - Statements prepared before the extension was negotiated (e.g., during a rolling | ||
| upgrade) retain ``result_metadata_id=None`` and fall back to always requesting | ||
| full metadata, which is the safest option. | ||
|
|
There was a problem hiding this comment.
Sure, but at some point we'll receive the metadata and id, and then it will be skipped.
I'm saying that because this point may suggest that clients need to be restarted after such rolling upgrade, but they don't
| directly from the data. | ||
|
|
||
| For full protocol details see the ScyllaDB CQL extensions documentation: | ||
| https://opensource.docs.scylladb.com/stable/cql/cql-extensions.html |
There was a problem hiding this comment.
What is this link? Opensource is LONG deprecated, and this link is about CQL language extension, not CQL protocol extensions - totally irrelevant here.
| if self.prepared_statement and isinstance(message, ExecuteMessage): | ||
| has_result_metadata_id = self.prepared_statement.result_metadata_id is not None | ||
| has_result_metadata = bool(self.prepared_statement.result_metadata) | ||
| can_skip_meta = has_result_metadata_id and has_result_metadata and ( | ||
| ProtocolVersion.uses_prepared_metadata(connection.protocol_version) |
There was a problem hiding this comment.
When is bool(self.prepared_statement.result_metadata) true?
Is it true if server sent result metadata with 0 columns?
Summary
Implements the
SCYLLA_USE_METADATA_IDScylla CQL protocol extension (DRIVER-153), which backports the prepared-statement metadata-ID mechanism from CQL v5 to earlier protocol versions.When the extension is negotiated:
skip_meta=True)METADATA_CHANGEDflag and includes the new metadata ID + new column metadata in the response — the driver picks this up and updates its cached metadata automaticallyChanges
cassandra/protocol_features.pyUSE_METADATA_ID = "SCYLLA_USE_METADATA_ID"constant anduse_metadata_idfield toProtocolFeaturescassandra/protocol.py_write_query_paramsnow actually writes_SKIP_METADATA_FLAGon the wire — it was stored on_QueryMessagebut never sent (effectively dead code)recv_results_prepared: readresult_metadata_idfor Scylla extension (pre-v5) in addition to standard CQL v5+ExecuteMessage: adduse_metadata_idflag (defaultFalse);send_bodygates theresult_metadata_idfield onProtocolVersion.uses_prepared_metadata(protocol_version) or self.use_metadata_id— both paths unified. An empty sentinel (b'') is written when the hash isNone(LWT / mixed cluster) so the frame layout is always correct and noTypeErrorcrash on v5cassandra/cluster.py_create_response_future: buildExecuteMessagewith safe defaults (skip_meta=False,result_metadata_id=None,use_metadata_id=False)_query: after borrowing the connection, setcan_skip_meta,skip_meta,result_metadata_id, anduse_metadata_idbased on the actual connection's negotiated features.skip_metais only enabled when the prepared statement has both aresult_metadata_idand usable cachedresult_metadata— guards against LWT/NO_METADATAstatements and mixed-cluster scenarios_set_result: on METADATA_CHANGED updateprepared_statement.result_metadatathenresult_metadata_idin that order (safe write ordering for concurrent readers)docs/scylla-specific.rstTest plan
_set_resultMETADATA_CHANGED path, and per-connection feature gating in_query(5 scenarios: extension on/off, metadata id present/absent, LWT, protocol v5)57 passedin affected files; 627+ overall)