Repository navigation
Conversation
Introduce the foundational window functions pipeline stage for Cloud Firestore: - Add `WindowFunction` base class (making `AggregateFunction` a subclass so existing aggregates can be used in window stages) and the `Rank` expression in `pipeline_expressions.py`. - Add `WindowSpec(partition=..., sort=...)` in `pipeline_types.py` and export it from `google.cloud.firestore` and `google.cloud.firestore_v1`. - Add `AddWindowFields` stage in `pipeline_stages.py` and `pipeline.add_window_fields()` on `_BasePipeline`. - Add unit tests and `window.yaml` acceptance test (`testAddWindowFields - rank with ties`), skipping live execution under #19115 until the backend rolls out to production.
| """A base class for window functions that operate over a window of documents.""" | ||
|
|
||
|
|
||
| class AggregateFunction(WindowFunction): |
There was a problem hiding this comment.
Design Decision (WindowFunction & AggregateFunction(WindowFunction)):
- Chosen Design: Introduce
WindowFunction(FunctionExpression)and makeAggregateFunctiona subclass ofWindowFunction. - Alternatives Considered & Why Rejected:
- Using
ExpressionorFunctionExpressiondirectly withoutWindowFunction: Rejected because arbitrary scalar expressions (likeField.of("x").add(1)) are not valid window functions, and when.over()is added in PR 4, putting.over()onExpressionwould allow invalid calls likeField.of("x").add(1).over(...). - Keeping
WindowFunctionandAggregateFunctionas separate sibling classes and using a unionWindowFunction | AggregateFunction: Rejected because every aggregate function (Sum,Average,Count, etc.) is a valid window function (whereasRank()is not valid inpipeline.aggregate()). Subclassing models this one-way relationship directly in the class hierarchy and means.over()in PR 4 only needs to be defined once onWindowFunction.
- Using
There was a problem hiding this comment.
This seems reasonable enough for the current functionality, but it looks like every other language is doing option ii, and keeping them as siblings. If a new feature comes down that assumes they are distinct classes, that would be hard to workaround later
I think we should stay consistent with other languages here. But you can create a private Mixin to hold shared logic without a direct inheritance relationship if that would be helpful
There was a problem hiding this comment.
Makes sense, switched AggregateFunction and WindowFunction to sibling subclasses of FunctionExpression so we don't risk a breaking change later and stay consistent with Java/Node.
| raise ValueError( | ||
| "AddWindowFields stage contains both positional and keyword fields" | ||
| ) | ||
| self.fields = list(args or fields) |
There was a problem hiding this comment.
Design Decision (add_window_fields & AddWindowFields Signature):
- Chosen Design:
pipeline.add_window_fields(*fields, window: WindowSpec | None = None)(expressions first as*fields, optionalwindowas a keyword argument). - Alternatives Considered & Why Rejected:
- Putting
windowas the first positional argument (add_window_fields(window, *fields), like Java/Web): Rejected for two reasons:- In Python's existing
pipeline.aggregate(*accumulators, groups=...), the expressions come first as*argsand the grouping config is a keyword argument at the end. - Because
windowis optional (when omitted, all documents form a single unpartitioned window), putting*fieldsfirst lets callers writepipeline.add_window_fields(Field.of("x").sum().as_("total"))without having to passWindowSpec()orNoneas a first argument.
- In Python's existing
- Why
AddWindowFields.__init__accepts both*argsandfields=(): Python callers pass expressions positionally (*args), while the YAML acceptance test runner (test_pipeline_acceptance.py) unpacksAddWindowFields: {window: ..., fields: [...]}fromwindow.yamlas**kwargs. Accepting either (and raisingValueErrorif both are passed at once) matchesAggregate.__init__right below.
- Putting
| def __eq__(self, other: object) -> bool: | ||
| if not isinstance(other, WindowSpec): | ||
| return NotImplemented | ||
| return self._to_pb() == other._to_pb() |
There was a problem hiding this comment.
Design Decision (WindowSpec):
- Chosen Design: Keyword-only constructor
WindowSpec(*, partition=None, sort=None)that accepts either a single value (partition="genre") or a sequence (partition=["genre", "author"]), and compares_to_pb()in__eq__. - Alternatives Considered & Why Rejected:
- Fluent builder methods (
WindowSpec().partition("genre").sort(...), like Java/Kotlin): Rejected because every existing stage options class in the Python Firestore SDK (SearchOptions,FindNearestOptions,UnnestOptions) uses keyword-argument constructors rather than fluent method chaining. - Passing
partitionandsortdirectly onadd_window_fields(..., partition=..., sort=...)withoutWindowSpec: Rejected because bundling window configuration intoWindowSpeckeepsadd_window_fieldsclean asdocuments=andrange=frames are added in PRs 2–3, and aligns with the cross-SDKWindowSpectype. - Positional arguments on
WindowSpec(partition, sort): Rejected in favor of keyword-only (*) so callers cannot accidentally mix up argument order oncedocuments=andrange=are added. - Comparing
self.sort == other.sortin__eq__: Rejected becauseOrderinginpipeline_types.pydoes not implement__eq__(so two identicalOrderinginstances compare asFalse). Comparingself._to_pb() == other._to_pb()(matchingSelectable.__eq__) makesWindowSpecequality work without touchingOrdering.
- Fluent builder methods (
| def _skip_window_execution_tests(request): | ||
| # TODO(#19115): Remove once add_window_fields is available in production. | ||
| if "window.yaml" in request.node.name and "parse_proto" not in request.node.name: | ||
| pytest.skip("Temporarily skipped. Not yet in production.") |
There was a problem hiding this comment.
Design Decision (Skipping Live window.yaml Execution Under #19115):
- Chosen Design: A single
@pytest.fixture(autouse=True)that skips live backend execution forwindow.yamlwhile keepingtest_pipeline_parse_protoactive (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115). - Alternatives Considered & Why Rejected:
- Commenting out
assert_resultsinwindow.yaml: Rejected because it leaves large commented-out YAML blocks across PRs 1–4. - Adding
if test_dict.get("file_name") == "window.yaml": pytest.skip(...)inside each of the 4 test functions: Rejected because it duplicates the skip check in 4 functions instead of 1 place.
- Commenting out
There was a problem hiding this comment.
I think we should discuss if we want to merge/release this before the backend is ready. This is a bit different than 16MB, where it's a silent upgrade. I think we have a couple options here:
A. Typically, we would merge approved PRs into a staging branch, and wait for prod to be ready.
B. It looks like some other languages are adding a note to the docstring saying "In preview until the backend feature launches". So we could also do that here, and add a task to come back to it when released
Let me know which one you'd prefer
There was a problem hiding this comment.
I'd prefer merging to main with a preview note so we don't have to maintain a staging branch across the stack. Updated the docstrings here (and will do the same for PRs 2-4), and unskipping the tests on launch is tracked in #19115. Let me know if that works for you.
There was a problem hiding this comment.
Code Review
This pull request introduces support for window functions in Firestore pipelines, adding the add_window_fields stage, WindowSpec configuration, and the Rank window function, along with corresponding unit and system tests. The review feedback identifies two important issues: a potential type-handling bug in WindowSpec's initialization where invalid sort arguments (such as strings or single expressions) are not properly validated, and an incorrect type annotation for window in base_pipeline.py that incorrectly references types.WindowSpec instead of the custom WindowSpec class.
| """A base class for window functions that operate over a window of documents.""" | ||
|
|
||
|
|
||
| class AggregateFunction(WindowFunction): |
There was a problem hiding this comment.
This seems reasonable enough for the current functionality, but it looks like every other language is doing option ii, and keeping them as siblings. If a new feature comes down that assumes they are distinct classes, that would be hard to workaround later
I think we should stay consistent with other languages here. But you can create a private Mixin to hold shared logic without a direct inheritance relationship if that would be helpful
| ... Field.of("price").sum().as_("genre_total_price"), | ||
| ... window=WindowSpec( | ||
| ... partition=["genre"], | ||
| ... sort=[Field.of("rating").descending()], |
There was a problem hiding this comment.
IIUC, having this sort here means that the value under genre_total_price is a running total, not the genre total
We should update the example, and make sure this sort detail is mentioned in this docstring, and in the WindowSpec
There was a problem hiding this comment.
Good catch! Updated the example to show both an unsorted partition average and a sorted rank + running total.
|
|
||
| Args: | ||
| *fields: Aliased `AggregateFunction` or `WindowFunction` expressions defining | ||
| the window calculations and their output field aliases. |
There was a problem hiding this comment.
It seems like other languages require at least one field. Should we do the same here?
There was a problem hiding this comment.
Kept *fields to match the other variadic stages in base_pipeline.py (add_fields, aggregate, select, sort, etc.), which all rely on backend validation. Happy to require a first arg here if you feel strongly though.
| *fields: Aliased `AggregateFunction` or `WindowFunction` expressions defining | ||
| the window calculations and their output field aliases. | ||
| window: Optional `WindowSpec` specifying the baseline window configuration | ||
| for this stage. |
There was a problem hiding this comment.
Can you explain what happens if window is None?
There was a problem hiding this comment.
Updated the docstring to document window=None (single unsorted global partition, equivalent to WindowSpec()).
| window: types.WindowSpec | None = None, | ||
| ) -> "_BasePipeline": | ||
| """ | ||
| Adds new fields computed by window functions over a window of documents. |
There was a problem hiding this comment.
nit: this docstring wasn't very helpful when I went to understand the feature. If this is how other lanuages handle this, it's probably fine though
There was a problem hiding this comment.
Updated the docstring to contrast with aggregate() and explain default frame behavior.
| """Creates a window function that computes the rank of each document within a window. | ||
|
|
||
| Documents with equal sort values receive the same rank, with gaps in the subsequent | ||
| ranking numbers. |
There was a problem hiding this comment.
Shouldn't we also include "The window must specify exactly one sort ordering"?
There was a problem hiding this comment.
Added! Tested on sandbox and 0 or 2+ sorts return 400.
| @@ -0,0 +1,102 @@ | |||
| tests: | |||
| - description: "testAddWindowFields - rank with ties" | |||
There was a problem hiding this comment.
Can we have some more test cases? Like using windowing with aggregate functions? different partitions? Maybe some error cases, like Rank without a sort?
There was a problem hiding this comment.
Added 3 tests (partition aggregates, sorted running window, and Rank without sort error) and verified against sandbox.
|
|
||
| Args: | ||
| partition: Field paths or expressions used to partition documents into independent windows. | ||
| sort: Orderings specifying how documents are sorted within each partition. |
There was a problem hiding this comment.
We should mention what happens if partition or sort is None
There was a problem hiding this comment.
Updated the docstring to document partition=None and sort=None.
| def _skip_window_execution_tests(request): | ||
| # TODO(#19115): Remove once add_window_fields is available in production. | ||
| if "window.yaml" in request.node.name and "parse_proto" not in request.node.name: | ||
| pytest.skip("Temporarily skipped. Not yet in production.") |
There was a problem hiding this comment.
I think we should discuss if we want to merge/release this before the backend is ready. This is a bit different than 16MB, where it's a silent upgrade. I think we have a couple options here:
A. Typically, we would merge approved PRs into a staging branch, and wait for prod to be ready.
B. It looks like some other languages are adding a note to the docstring saying "In preview until the backend feature launches". So we could also do that here, and add a task to come back to it when released
Let me know which one you'd prefer
- Make AggregateFunction and WindowFunction sibling subclasses of FunctionExpression and accept both in add_window_fields/AddWindowFields - Validate WindowSpec sort inputs and add unit test coverage - Update docstrings with preview notes, window=None, default frame semantics, Rank single-sort requirement, and fixed running-total example - Add partition aggregate, running window, and Rank error tests to window.yaml
Goal
Add the foundational
add_window_fieldspipeline stage to the Cloud Firestore Python SDK, including:AddWindowFieldsstage andpipeline.add_window_fields(*fields, window=None)method (for both syncPipelineandAsyncPipeline).WindowSpec(*, partition=None, sort=None)for partitioning documents into independent buckets (partition) and ordering documents within each bucket (sort).WindowFunctionbase class (withAggregateFunctioninheriting fromWindowFunction) and theRank()window expression.Background: What Are Window Functions?
In Firestore Pipelines today,
pipeline.aggregate(...)groups documents together and collapses them into summary rows (for example, turning 100 book documents into 2 summary rows—one per genre).Window functions (
pipeline.add_window_fields(...)) let users compute aggregates (like a genre average or running total) or rankings (Rank()) without collapsing the documents—every original document is preserved, with the computed window values attached as new fields on each document (similar toOVER (PARTITION BY ... ORDER BY ...)in SQL or$setWindowFieldsin MongoDB).Concrete Example
Suppose a
"books"collection has these 4 documents:titlegenrerating"Dune""Sci-Fi"4.6"1984""Sci-Fi"4.2"Hitchhiker's Guide""Sci-Fi"4.2"Handmaid's Tale""Sci-Fi"4.1Every book document is kept, and two new fields (
genre_avg_ratingandrating_rank) are added to each document (with tied ratings receiving the same rank and skipping the next rank):titlegenreratinggenre_avg_ratingrating_rank"Handmaid's Tale""Sci-Fi"4.14.11"1984""Sci-Fi"4.24.1672(tie)"Hitchhiker's Guide""Sci-Fi"4.24.1672(tie)"Dune""Sci-Fi"4.64.2754(skips 3)Summary of Changes
pipeline_expressions.py):WindowFunction(FunctionExpression)as the base class for all window functions.AggregateFunctioninherit fromWindowFunctionso all existing aggregate expressions (Sum,Average,Count,Min,Max, etc.) can be used inadd_window_fields, while keeping window-only functions likeRankdisallowed inpipeline.aggregate().Rank()window function expression.pipeline_types.py):WindowSpec(*, partition=None, sort=None)and export it fromgoogle.cloud.firestoreandgoogle.cloud.firestore_v1(including.librarian/generator-input/client-post-processing/firestore-integration.yaml).pipeline_stages.py,base_pipeline.py):AddWindowFieldsstage class andpipeline.add_window_fields(*fields, window=None)builder method.tests/unit/v1/,tests/system/):test_pipeline_stages.py,test_pipeline_expressions.py,test_pipeline.py, andtest_async_pipeline.py.tests/system/pipeline_e2e/window.yaml(testAddWindowFields - rank with ties) and skip live execution in CI via_skip_window_execution_tests(firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115) while keepingtest_pipeline_parse_protoactive.What We Will Follow Up On
To keep each PR small and easy to review, the rest of the window functions feature is split into follow-up PRs:
WindowBound,DocumentWindowFrame(preceding=..., following=...), and thedocuments=parameter onWindowSpecfor row-offset sliding windows (e.g., "1 row before to 1 row after").RangeWindowFrame(preceding=..., following=..., unit=None), therange=parameter onWindowSpecfor value/time-offset sliding windows (e.g., "within the last 20 years"), and mutual-exclusivity validation when bothdocumentsandrangeare passed.WindowFunction.over(frame)so individual window functions insideadd_window_fieldscan define their ownDocumentWindowFrameorRangeWindowFrameinstead of sharing the stage-level frame.Expression.as_()(Standalone Follow-up): UpdateExpression.as_()andAliasedExpression[T]covariance in a separate PR so static type checkers preserve the inner subtype (AliasedExpression[WindowFunction]/AliasedExpression[AggregateFunction]) when.as_("alias")is called on an expression.window.yamlExecution in CI (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115): Remove the temporary_skip_window_execution_testsfixture intests/system/test_pipeline_acceptance.pyonceadd_window_fieldsrolls out to production.Verification
pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py(468 passed)test_pipeline_parse_proto,test_pipeline_results, andtest_pipeline_results_asyncpass againsttest-firestore.sandbox.googleapis.com:443; verified live execution tests skip cleanly in CI mode (1 passed, 2 skipped).ruff check --select I,ruff format --check, andflake8pass.