Repository navigation
Conversation
… windows Introduce WindowBound, _WindowFrame, and DocumentWindowFrame in pipeline_types.py and add the documents parameter to WindowSpec to support document-count sliding window frames (e.g., running totals and moving averages) in AddWindowFields stages. Export DocumentWindowFrame and WindowBound from google.cloud.firestore and google.cloud.firestore_v1, and add unit and window.yaml system tests.
| @dataclass | ||
| class DocumentWindowFrame(_WindowFrame): | ||
| """Defines a document-count based window frame relative to the current document. | ||
|
|
||
| Example: | ||
| >>> from google.cloud.firestore_v1.pipeline_types import ( | ||
| ... DocumentWindowFrame, | ||
| ... WindowBound, | ||
| ... WindowSpec, | ||
| ... ) | ||
| >>> window = WindowSpec( | ||
| ... documents=DocumentWindowFrame( | ||
| ... WindowBound.UNBOUNDED, WindowBound.CURRENT | ||
| ... ) | ||
| ... ) | ||
|
|
||
| Args: | ||
| preceding: The lower bound of the frame (an integer offset, `WindowBound`, or `Expression`). | ||
| A positive integer includes up to that many documents before the current document. | ||
| A negative integer indicates a boundary following the current document. | ||
| following: The upper bound of the frame (an integer offset, `WindowBound`, or `Expression`). | ||
| A positive integer includes up to that many documents after the current document. | ||
| A negative integer indicates a boundary preceding the current document. | ||
| """ | ||
|
|
||
| preceding: WindowBound | str | int | pipeline_expressions.Expression | ||
| following: WindowBound | str | int | pipeline_expressions.Expression |
There was a problem hiding this comment.
Design Note — DocumentWindowFrame Dataclass & _to_dict():
- Chosen Design:
DocumentWindowFrame(preceding, following)is a standalone@dataclass(inheriting_to_pb()from_WindowFrameand exposing_to_dict()->{"documents": self._to_pb()}) passed toWindowSpec(frame=...), with symbolic bounds represented byWindowBound(str, Enum)(matchingTimeUnit/PipelineDataTypein this module). - Alternatives Considered & Why Rejected:
- Builder method on
WindowSpec(e.g.,WindowSpec().documents(preceding, following)as in Java): Python pipeline option types (FindNearestOptions,SearchOptions,UnnestOptions) use keyword-argument constructors rather than fluent builders. A standalone frame class is also reusable directly in per-function.over(DocumentWindowFrame(...))overrides in a follow-up PR. - Raw
tupleordict(frame=(-1, "current")orframe={"preceding": -1, "following": "current"}): A tuple hides which bound isprecedingvsfollowingand doesn't extend cleanly whenRangeWindowFrameaddsunit=...; adictloses type annotations and autocomplete.
- Builder method on
There was a problem hiding this comment.
Code Review
This pull request introduces support for document-count based window frames in Firestore pipeline queries by adding WindowBound and DocumentWindowFrame classes, and integrating them into WindowSpec. The review feedback correctly identifies a critical bug where the __eq__ method of WindowSpec was not updated to include the new documents attribute, which would lead to incorrect equality evaluations. A code suggestion is provided to resolve this issue.
…l librarian post-processing
Replace WindowSpec(documents=...) with WindowSpec(frame=...) so nominal window frame types use a single unified parameter without repeating the frame kind. Add _to_dict() on DocumentWindowFrame to emit its wire map entry directly.
| | str | ||
| ] = None, | ||
| sort: Optional[Sequence[Ordering] | Ordering] = None, | ||
| frame: Optional[DocumentWindowFrame] = None, |
There was a problem hiding this comment.
Design Note — WindowSpec(frame=...) & Backend-Enforced sort Validation:
- Chosen Design:
WindowSpecaccepts a singleframe=parameter (widened toDocumentWindowFrame | RangeWindowFramein the next PR) and delegates wire-key dispatch viaself.frame._to_dict().WindowSpec.__init__does not raise a client-sideValueErrorwhenframeis provided withoutsort.
- Alternatives Considered & Why Rejected:
- Separate
documents=andrange=keyword arguments (as in Node's plain-objectOneOf): In Python, the frame argument is already a nominally typed class (DocumentWindowFramevsRangeWindowFrame), so repeating the frame kind in the keyword name (documents=DocumentWindowFrame(...),range=RangeWindowFrame(...)) stutters, shadows built-inrange, and requires runtime mutual-exclusivity validation (if documents is not None and range is not None: raise ValueError(...)). A singleframe=parameter makes passing two frames syntactically impossible and matchesWindowFunction.over(frame). - Raising
ValueErrorinWindowSpec.__init__whenframeis set andsortis omitted: Rejected per the cross-SDK Window Functions spec—sortrequirement checks are intentionally enforced by the backend at execution time rather than inWindowSpec.__init__(which also avoids duplicating backend validation rules when per-function.over(frame)overrides are used).
- Separate
There was a problem hiding this comment.
This seems reasonable to me, but the fact that all the other libraries use a different pattern makes me a bit nervous. WindowSpec(documents=DocumentWindowFrame(1, "current")) seems more consistent with others, and I don't think a mutual exclusion check is too much of a burden. Maybe types could be simplified more if you wanted to avoid the naming stutter
But that said, I can't think of any real concerns with this current approach, and it does seem cleaner/nicer to me in isolation. So LGTM if you want to keep it as-is
| >>> window = WindowSpec( | ||
| ... sort=Field.of("published").ascending(), | ||
| ... frame=DocumentWindowFrame( | ||
| ... WindowBound.UNBOUNDED, WindowBound.CURRENT |
There was a problem hiding this comment.
nit: the example could be more clear with preceding and following kwargs present
| "partition=[Field.of('genre')], " | ||
| "sort=[Field.of('rating').descending()]" | ||
| "sort=[Field.of('rating').descending()], " | ||
| "frame=DocumentWindowFrame(preceding=-1, following=<WindowBound.CURRENT: 'current'>)" |
There was a problem hiding this comment.
Can we fix the enum repr for <WindowBound.CURRENT: 'current'>?
| spec = WindowSpec( | ||
| partition="genre", | ||
| sort=Field.of("rating").descending(), | ||
| frame=DocumentWindowFrame(-1, WindowBound.CURRENT), |
There was a problem hiding this comment.
Doesn't -1 mean "a boundary following the current document" here? So isn't this boundary backwards, where preceding is after following?
| Args: | ||
| partition: Field paths or expressions used to partition documents into independent windows. | ||
| sort: Orderings specifying how documents are sorted within each partition. | ||
| frame: A `DocumentWindowFrame` defining a document-count window frame. |
There was a problem hiding this comment.
What happens if None?
| @dataclass | ||
| class DocumentWindowFrame(_WindowFrame): | ||
| """Defines a document-count based window frame relative to the current document. | ||
|
|
There was a problem hiding this comment.
A bit more context to include:
Bounds count documents, not values. `WindowBound.CURRENT` refers only to the
current document's position, so documents with the same sort value are not
automatically included in the frame.
`sort` on the enclosing `WindowSpec` is optional for a document frame. If it
is omitted, documents are processed in the order they reach the stage.
| frame: | ||
| DocumentWindowFrame: | ||
| preceding: unbounded | ||
| following: current |
There was a problem hiding this comment.
can we have a test with a numeric offset? (Maybe ask gemini for more test cases too)
| | str | ||
| ] = None, | ||
| sort: Optional[Sequence[Ordering] | Ordering] = None, | ||
| frame: Optional[DocumentWindowFrame] = None, |
There was a problem hiding this comment.
This seems reasonable to me, but the fact that all the other libraries use a different pattern makes me a bit nervous. WindowSpec(documents=DocumentWindowFrame(1, "current")) seems more consistent with others, and I don't think a mutual exclusion check is too much of a burden. Maybe types could be simplified more if you wanted to avoid the naming stutter
But that said, I can't think of any real concerns with this current approach, and it does seem cleaner/nicer to me in isolation. So LGTM if you want to keep it as-is
Goal
Add document-count sliding window frames (
DocumentWindowFrameandWindowBound) toWindowSpec, building onadd_window_fields()from #19116.Background: What Are Document Window Frames?
In #19116,
WindowSpec(partition=..., sort=...)lets us partition and order documents, using the backend's default window frame:sort): Includes all documents in the partition (preceding="unbounded",following="unbounded").sort): Includes documents from the start of the partition up to the current document's sort value (preceding="unbounded",following="current").A document window frame (
frame=DocumentWindowFrame(preceding, following)) lets the caller explicitly control how many neighboring documents before (preceding) and after (following) the current document are included in the window calculation (similar to SQLROWS BETWEEN ... AND ...):WindowBound.UNBOUNDED("unbounded"): Extends to the first (preceding) or last (following) document in the partition.WindowBound.CURRENT("current"): Starts or ends at the current document.intorExpression):preceding: a positive integerfollowing: a positive integerFor example,
DocumentWindowFrame(WindowBound.UNBOUNDED, WindowBound.CURRENT)computes a running total and running count from the first book in each genre up to the current book:titlegenrepublishedratingrunning_rating(sum)running_count(Count)"1984""Dystopian""The Handmaid's Tale""Dystopian""Dune""Science Fiction""The Hitchhiker's Guide to the Galaxy""Science Fiction"Summary of Changes
pipeline_types.py):WindowBound(str, Enum)withCURRENT = "current"andUNBOUNDED = "unbounded"._WindowFramebase dataclass andDocumentWindowFrame(preceding, following)dataclass with_to_dict().frame: Optional[DocumentWindowFrame] = NonetoWindowSpec.firestore/__init__.py,firestore_v1/__init__.py,firestore-integration.yaml):DocumentWindowFrameandWindowBound.DocumentWindowFrameandWindowSpec(frame=...)intests/unit/v1/test_pipeline_stages.py.testAddWindowFields - partition, sort, documents frame, and multiple accumulatorsintests/system/pipeline_e2e/window.yamland updatetests/system/test_pipeline_acceptance.pyto resolvepipeline_typesclasses from YAML.What We Will Follow Up On
RangeWindowFrame): Follow-up PR addingRangeWindowFrame(preceding, following, unit=None)and wideningWindowSpec(frame=...)to acceptDocumentWindowFrame | RangeWindowFrame..over(frame)): Follow-up PR addingWindowFunction.over(frame)so individual accumulators can override the stage's frame._skip_window_execution_testsintests/system/test_pipeline_acceptance.pyonce theadd_window_fieldsbackend rollout is live in production.Verification
pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py(469 passed, 100% statement/branch coverage onpipeline_types.py).pytest tests/system/test_pipeline_acceptance.py -k "window.yaml"(verified both proto serialization and live sync/async execution againsttest-firestore.sandbox.googleapis.com:443: 6 passed).mypy,ruff check --select I,ruff format --check, andflake8 google testsall pass cleanly.