Skip to content

feat(firestore): add AddWindowFields stage, WindowSpec, and Rank - #19116

Open
ohmayr wants to merge 2 commits into
mainfrom
feat/firestore-window-1-stage
Open

ohmayr wants to merge 2 commits into
mainfrom
feat/firestore-window-1-stage

Conversation

@ohmayr

@ohmayr ohmayr commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Goal

Add the foundational add_window_fields pipeline stage to the Cloud Firestore Python SDK, including:

  1. The AddWindowFields stage and pipeline.add_window_fields(*fields, window=None) method (for both sync Pipeline and AsyncPipeline).
  2. WindowSpec(*, partition=None, sort=None) for partitioning documents into independent buckets (partition) and ordering documents within each bucket (sort).
  3. The WindowFunction base class (with AggregateFunction inheriting from WindowFunction) and the Rank() window expression.

Background: What Are Window Functions?

In Firestore Pipelines today, pipeline.aggregate(...) groups documents together and collapses them into summary rows (for example, turning 100 book documents into 2 summary rows—one per genre).

Window functions (pipeline.add_window_fields(...)) let users compute aggregates (like a genre average or running total) or rankings (Rank()) without collapsing the documents—every original document is preserved, with the computed window values attached as new fields on each document (similar to OVER (PARTITION BY ... ORDER BY ...) in SQL or $setWindowFields in MongoDB).

Concrete Example

Suppose a "books" collection has these 4 documents:

title genre rating
"Dune" "Sci-Fi" 4.6
"1984" "Sci-Fi" 4.2
"Hitchhiker's Guide" "Sci-Fi" 4.2
"Handmaid's Tale" "Sci-Fi" 4.1
from google.cloud.firestore import Client, WindowSpec
from google.cloud.firestore_v1.pipeline_expressions import Field, Rank

client = Client()
pipeline = (
    client.pipeline()
    .collection("books")
    .add_window_fields(
        Field.of("rating").average().as_("genre_avg_rating"),
        Rank().as_("rating_rank"),
        window=WindowSpec(
            partition="genre",
            sort=Field.of("rating").ascending(),
        ),
    )
)

Every book document is kept, and two new fields (genre_avg_rating and rating_rank) are added to each document (with tied ratings receiving the same rank and skipping the next rank):

title genre rating genre_avg_rating rating_rank
"Handmaid's Tale" "Sci-Fi" 4.1 4.1 1
"1984" "Sci-Fi" 4.2 4.167 2 (tie)
"Hitchhiker's Guide" "Sci-Fi" 4.2 4.167 2 (tie)
"Dune" "Sci-Fi" 4.6 4.275 4 (skips 3)

Summary of Changes

  • Window Expression Hierarchy (pipeline_expressions.py):
    • Add WindowFunction(FunctionExpression) as the base class for all window functions.
    • Make AggregateFunction inherit from WindowFunction so all existing aggregate expressions (Sum, Average, Count, Min, Max, etc.) can be used in add_window_fields, while keeping window-only functions like Rank disallowed in pipeline.aggregate().
    • Add the Rank() window function expression.
  • Window Configuration (pipeline_types.py):
    • Add WindowSpec(*, partition=None, sort=None) and export it from google.cloud.firestore and google.cloud.firestore_v1 (including .librarian/generator-input/client-post-processing/firestore-integration.yaml).
  • Pipeline Stage (pipeline_stages.py, base_pipeline.py):
    • Add the AddWindowFields stage class and pipeline.add_window_fields(*fields, window=None) builder method.
  • Tests (tests/unit/v1/, tests/system/):

What We Will Follow Up On

To keep each PR small and easy to review, the rest of the window functions feature is split into follow-up PRs:

  1. Document Window Frames (PR 2): Add WindowBound, DocumentWindowFrame(preceding=..., following=...), and the documents= parameter on WindowSpec for row-offset sliding windows (e.g., "1 row before to 1 row after").
  2. Range Window Frames (PR 3): Add RangeWindowFrame(preceding=..., following=..., unit=None), the range= parameter on WindowSpec for value/time-offset sliding windows (e.g., "within the last 20 years"), and mutual-exclusivity validation when both documents and range are passed.
  3. Per-Function Frame Overrides (PR 4): Add WindowFunction.over(frame) so individual window functions inside add_window_fields can define their own DocumentWindowFrame or RangeWindowFrame instead of sharing the stage-level frame.
  4. Type-Checker Preservation on Expression.as_() (Standalone Follow-up): Update Expression.as_() and AliasedExpression[T] covariance in a separate PR so static type checkers preserve the inner subtype (AliasedExpression[WindowFunction] / AliasedExpression[AggregateFunction]) when .as_("alias") is called on an expression.
  5. Enable Live window.yaml Execution in CI (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115): Remove the temporary _skip_window_execution_tests fixture in tests/system/test_pipeline_acceptance.py once add_window_fields rolls out to production.

Verification

  • Unit Tests: pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py (468 passed)
  • System Tests (Nightly Sandbox Endpoint): Verified test_pipeline_parse_proto, test_pipeline_results, and test_pipeline_results_async pass against test-firestore.sandbox.googleapis.com:443; verified live execution tests skip cleanly in CI mode (1 passed, 2 skipped).
  • Lint: ruff check --select I, ruff format --check, and flake8 pass.

Introduce the foundational window functions pipeline stage for Cloud
Firestore:
- Add `WindowFunction` base class (making `AggregateFunction` a
  subclass so existing aggregates can be used in window stages) and
  the `Rank` expression in `pipeline_expressions.py`.
- Add `WindowSpec(partition=..., sort=...)` in `pipeline_types.py`
  and export it from `google.cloud.firestore` and
  `google.cloud.firestore_v1`.
- Add `AddWindowFields` stage in `pipeline_stages.py` and
  `pipeline.add_window_fields()` on `_BasePipeline`.
- Add unit tests and `window.yaml` acceptance test (`testAddWindowFields
  - rank with ties`), skipping live execution under #19115 until the
  backend rolls out to production.
@ohmayr
ohmayr requested review from a team as code owners November 18, 2026 01:04
"""A base class for window functions that operate over a window of documents."""


class AggregateFunction(WindowFunction):

@ohmayr ohmayr Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Decision (WindowFunction & AggregateFunction(WindowFunction)):

  • Chosen Design: Introduce WindowFunction(FunctionExpression) and make AggregateFunction a subclass of WindowFunction.
  • Alternatives Considered & Why Rejected:
    1. Using Expression or FunctionExpression directly without WindowFunction: Rejected because arbitrary scalar expressions (like Field.of("x").add(1)) are not valid window functions, and when .over() is added in PR 4, putting .over() on Expression would allow invalid calls like Field.of("x").add(1).over(...).
    2. Keeping WindowFunction and AggregateFunction as separate sibling classes and using a union WindowFunction | AggregateFunction: Rejected because every aggregate function (Sum, Average, Count, etc.) is a valid window function (whereas Rank() is not valid in pipeline.aggregate()). Subclassing models this one-way relationship directly in the class hierarchy and means .over() in PR 4 only needs to be defined once on WindowFunction.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable enough for the current functionality, but it looks like every other language is doing option ii, and keeping them as siblings. If a new feature comes down that assumes they are distinct classes, that would be hard to workaround later

I think we should stay consistent with other languages here. But you can create a private Mixin to hold shared logic without a direct inheritance relationship if that would be helpful

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, switched AggregateFunction and WindowFunction to sibling subclasses of FunctionExpression so we don't risk a breaking change later and stay consistent with Java/Node.

raise ValueError(
"AddWindowFields stage contains both positional and keyword fields"
)
self.fields = list(args or fields)

@ohmayr ohmayr Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Decision (add_window_fields & AddWindowFields Signature):

  • Chosen Design: pipeline.add_window_fields(*fields, window: WindowSpec | None = None) (expressions first as *fields, optional window as a keyword argument).
  • Alternatives Considered & Why Rejected:
    1. Putting window as the first positional argument (add_window_fields(window, *fields), like Java/Web): Rejected for two reasons:
      • In Python's existing pipeline.aggregate(*accumulators, groups=...), the expressions come first as *args and the grouping config is a keyword argument at the end.
      • Because window is optional (when omitted, all documents form a single unpartitioned window), putting *fields first lets callers write pipeline.add_window_fields(Field.of("x").sum().as_("total")) without having to pass WindowSpec() or None as a first argument.
    2. Why AddWindowFields.__init__ accepts both *args and fields=(): Python callers pass expressions positionally (*args), while the YAML acceptance test runner (test_pipeline_acceptance.py) unpacks AddWindowFields: {window: ..., fields: [...]} from window.yaml as **kwargs. Accepting either (and raising ValueError if both are passed at once) matches Aggregate.__init__ right below.

def __eq__(self, other: object) -> bool:
if not isinstance(other, WindowSpec):
return NotImplemented
return self._to_pb() == other._to_pb()

@ohmayr ohmayr Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Decision (WindowSpec):

  • Chosen Design: Keyword-only constructor WindowSpec(*, partition=None, sort=None) that accepts either a single value (partition="genre") or a sequence (partition=["genre", "author"]), and compares _to_pb() in __eq__.
  • Alternatives Considered & Why Rejected:
    1. Fluent builder methods (WindowSpec().partition("genre").sort(...), like Java/Kotlin): Rejected because every existing stage options class in the Python Firestore SDK (SearchOptions, FindNearestOptions, UnnestOptions) uses keyword-argument constructors rather than fluent method chaining.
    2. Passing partition and sort directly on add_window_fields(..., partition=..., sort=...) without WindowSpec: Rejected because bundling window configuration into WindowSpec keeps add_window_fields clean as documents= and range= frames are added in PRs 2–3, and aligns with the cross-SDK WindowSpec type.
    3. Positional arguments on WindowSpec(partition, sort): Rejected in favor of keyword-only (*) so callers cannot accidentally mix up argument order once documents= and range= are added.
    4. Comparing self.sort == other.sort in __eq__: Rejected because Ordering in pipeline_types.py does not implement __eq__ (so two identical Ordering instances compare as False). Comparing self._to_pb() == other._to_pb() (matching Selectable.__eq__) makes WindowSpec equality work without touching Ordering.

def _skip_window_execution_tests(request):
# TODO(#19115): Remove once add_window_fields is available in production.
if "window.yaml" in request.node.name and "parse_proto" not in request.node.name:
pytest.skip("Temporarily skipped. Not yet in production.")

@ohmayr ohmayr Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Decision (Skipping Live window.yaml Execution Under #19115):

  • Chosen Design: A single @pytest.fixture(autouse=True) that skips live backend execution for window.yaml while keeping test_pipeline_parse_proto active (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115).
  • Alternatives Considered & Why Rejected:
    1. Commenting out assert_results in window.yaml: Rejected because it leaves large commented-out YAML blocks across PRs 1–4.
    2. Adding if test_dict.get("file_name") == "window.yaml": pytest.skip(...) inside each of the 4 test functions: Rejected because it duplicates the skip check in 4 functions instead of 1 place.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should discuss if we want to merge/release this before the backend is ready. This is a bit different than 16MB, where it's a silent upgrade. I think we have a couple options here:

A. Typically, we would merge approved PRs into a staging branch, and wait for prod to be ready.
B. It looks like some other languages are adding a note to the docstring saying "In preview until the backend feature launches". So we could also do that here, and add a task to come back to it when released

Let me know which one you'd prefer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer merging to main with a preview note so we don't have to maintain a staging branch across the stack. Updated the docstrings here (and will do the same for PRs 2-4), and unskipping the tests on launch is tracked in #19115. Let me know if that works for you.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for window functions in Firestore pipelines, adding the add_window_fields stage, WindowSpec configuration, and the Rank window function, along with corresponding unit and system tests. The review feedback identifies two important issues: a potential type-handling bug in WindowSpec's initialization where invalid sort arguments (such as strings or single expressions) are not properly validated, and an incorrect type annotation for window in base_pipeline.py that incorrectly references types.WindowSpec instead of the custom WindowSpec class.

"""A base class for window functions that operate over a window of documents."""


class AggregateFunction(WindowFunction):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable enough for the current functionality, but it looks like every other language is doing option ii, and keeping them as siblings. If a new feature comes down that assumes they are distinct classes, that would be hard to workaround later

I think we should stay consistent with other languages here. But you can create a private Mixin to hold shared logic without a direct inheritance relationship if that would be helpful

... Field.of("price").sum().as_("genre_total_price"),
... window=WindowSpec(
... partition=["genre"],
... sort=[Field.of("rating").descending()],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, having this sort here means that the value under genre_total_price is a running total, not the genre total

We should update the example, and make sure this sort detail is mentioned in this docstring, and in the WindowSpec

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Updated the example to show both an unsorted partition average and a sorted rank + running total.


Args:
*fields: Aliased `AggregateFunction` or `WindowFunction` expressions defining
the window calculations and their output field aliases.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like other languages require at least one field. Should we do the same here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept *fields to match the other variadic stages in base_pipeline.py (add_fields, aggregate, select, sort, etc.), which all rely on backend validation. Happy to require a first arg here if you feel strongly though.

*fields: Aliased `AggregateFunction` or `WindowFunction` expressions defining
the window calculations and their output field aliases.
window: Optional `WindowSpec` specifying the baseline window configuration
for this stage.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what happens if window is None?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the docstring to document window=None (single unsorted global partition, equivalent to WindowSpec()).

window: types.WindowSpec | None = None,
) -> "_BasePipeline":
"""
Adds new fields computed by window functions over a window of documents.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this docstring wasn't very helpful when I went to understand the feature. If this is how other lanuages handle this, it's probably fine though

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the docstring to contrast with aggregate() and explain default frame behavior.

"""Creates a window function that computes the rank of each document within a window.

Documents with equal sort values receive the same rank, with gaps in the subsequent
ranking numbers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also include "The window must specify exactly one sort ordering"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added! Tested on sandbox and 0 or 2+ sorts return 400.

@@ -0,0 +1,102 @@
tests:
- description: "testAddWindowFields - rank with ties"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have some more test cases? Like using windowing with aggregate functions? different partitions? Maybe some error cases, like Rank without a sort?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 3 tests (partition aggregates, sorted running window, and Rank without sort error) and verified against sandbox.


Args:
partition: Field paths or expressions used to partition documents into independent windows.
sort: Orderings specifying how documents are sorted within each partition.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should mention what happens if partition or sort is None

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the docstring to document partition=None and sort=None.

def _skip_window_execution_tests(request):
# TODO(#19115): Remove once add_window_fields is available in production.
if "window.yaml" in request.node.name and "parse_proto" not in request.node.name:
pytest.skip("Temporarily skipped. Not yet in production.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should discuss if we want to merge/release this before the backend is ready. This is a bit different than 16MB, where it's a silent upgrade. I think we have a couple options here:

A. Typically, we would merge approved PRs into a staging branch, and wait for prod to be ready.
B. It looks like some other languages are adding a note to the docstring saying "In preview until the backend feature launches". So we could also do that here, and add a task to come back to it when released

Let me know which one you'd prefer

- Make AggregateFunction and WindowFunction sibling subclasses of
  FunctionExpression and accept both in add_window_fields/AddWindowFields
- Validate WindowSpec sort inputs and add unit test coverage
- Update docstrings with preview notes, window=None, default frame
  semantics, Rank single-sort requirement, and fixed running-total
  example
- Add partition aggregate, running window, and Rank error tests to
  window.yaml

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants