Repository navigation
Conversation
Add WindowFunction.over(frame) to allow per-accumulator document or range window frame overrides inside an AddWindowFields stage. Wrap the frame argument in an internal _WindowFrameExpression dataclass adapter (mirroring _PipelineValueExpression) so FunctionExpression stays generic and WindowFunction requires no custom dunder or protobuf overrides. Add unit tests and pipeline_e2e/window.yaml system tests covering accumulator-level frames and backend rejection of Rank().over(...).
| class WindowFunction(FunctionExpression): | ||
| """A base class for window functions that operate over a window of documents.""" | ||
|
|
||
| def over( |
There was a problem hiding this comment.
Chosen Design: Define over(frame) directly on WindowFunction (inherited by AggregateFunction, which subclasses WindowFunction), returning a WindowFunction and letting the backend reject invalid pairings like Rank().over(frame) (verified in window.yaml).
Alternatives Considered & Why Rejected:
- Placing
over(frame)only onAggregateFunction: Excludes upcoming non-aggregate window functions that do accept frames (e.g.,first_value,last_value) or requires a third intermediate class solely for framing. - Client-side rejection of
Rank().over(frame): Duplicates backend validation and risks drifting as window function support evolves; aligns with Node.js and Java.
|
|
||
|
|
||
| @dataclass | ||
| class _WindowFrameExpression(Expression): |
There was a problem hiding this comment.
Chosen Design: Wrap frame in a small @dataclass Expression adapter (_WindowFrameExpression, mirroring _PipelineValueExpression directly above) so over() constructs WindowFunction("over", [self, _WindowFrameExpression(frame)], repr_function=...). This keeps FunctionExpression untouched (params: Sequence[Expression]), avoids importing WindowSpec here, gives WindowFunction zero dunder/_to_pb boilerplate, and lets the backend reject chained .over(f1).over(f2) (over(over(...))) rather than silently overwriting f1.
Alternatives Considered & Why Rejected:
- Passing
WindowSpec(frame=frame)and wideningFunctionExpression.params: Leaks window-stage configuration types into the genericFunctionExpressionbase class. - Storing
self._frameonWindowFunctionwith custom__init__,__repr__,__eq__,_to_pb: Adds four overrides toWindowFunctionsolely forover(), and risks silently droppingf1if chained. - Dedicated
_OverExpression(WindowFunction)subclass: Still introduces an internal class while requiring manual__init__,__eq__, and_to_pbinstead of reusingFunctionExpression+@dataclass.
There was a problem hiding this comment.
Code Review
This pull request introduces the over method to the WindowFunction class in pipeline_expressions.py, enabling the configuration of window frames (DocumentWindowFrame or RangeWindowFrame) for window functions. It also implements an internal _WindowFrameExpression wrapper to serialize these frames as expressions, and adds comprehensive unit and system tests to validate the new functionality. There are no review comments, so I have no feedback to provide.
Goal
Add per-accumulator window frame overrides via
WindowFunction.over(frame), building onDocumentWindowFrame(#19117) andRangeWindowFrame(#19119).Background: Per-Accumulator Window Frames
Setting
frame=...onWindowSpecapplies one default frame to every accumulator in anAddWindowFieldsstage. Often, however, a single partition and sort order needs to evaluate multiple accumulators over different frames—for example, computing a 20-year trailing count alongside a 3-document sliding window of nearby titles in one pass:On the wire,
fn.over(frame)encodes as a function callValue(function_value={"name": "over", "args": [fn._to_pb(), Value(map_value={"fields": frame._to_dict()})]}).Summary of Changes
WindowFunction.over(frame: DocumentWindowFrame | RangeWindowFrame) -> WindowFunctioningoogle/cloud/firestore_v1/pipeline_expressions.py(inherited byAggregateFunction, so all aggregate functions support.over(...))._WindowFrameExpression(Expression)@dataclassadapter (mirroring_PipelineValueExpression) soFunctionExpressionstays generic andWindowFunctionrequires no custom dunder or protobuf overrides.tests/unit/v1/test_pipeline_expressions.pycoveringrepr, equality, type hierarchy (not hasattr(Field.of("rating"), "over")), and protobuf serialization for both document and range frames.testAddWindowFields - accumulator-level over() framesandtestAddWindowFields - error on rank() with over() frametotests/system/pipeline_e2e/window.yaml.What We Will Follow Up On
window.yamlsystem execution tests in CI (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115) once theadd_window_fieldsbackend rollout is live in production.Verification
pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py(475 passed; 0 missed lines or branches across all added lines inpipeline_expressions.py, 100% coverage onpipeline_types.py).pytest tests/system/test_pipeline_acceptance.py -k window.yaml(proto serialization check in default CI mode, plus live sync/async execution againsttest-firestore.sandbox.googleapis.com:443: 16 passed).mypy,ruff check --select I,ruff format --check, andflake8 google testsall pass cleanly.