Skip to content

feat(firestore): add DocumentWindowFrame and WindowBound for pipeline windows - #19117

Open
ohmayr wants to merge 4 commits into
feat/firestore-window-1-stagefrom
feat/firestore-window-2-document-frame
Open

ohmayr wants to merge 4 commits into
feat/firestore-window-1-stagefrom
feat/firestore-window-2-document-frame

Conversation

@ohmayr

@ohmayr ohmayr commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Goal

Add document-count sliding window frames (DocumentWindowFrame and WindowBound) to WindowSpec, building on add_window_fields() from #19116.

Background: What Are Document Window Frames?

In #19116, WindowSpec(partition=..., sort=...) lets us partition and order documents, using the backend's default window frame:

  • Partition-only window (no sort): Includes all documents in the partition (preceding="unbounded", following="unbounded").
  • Sorted window (with sort): Includes documents from the start of the partition up to the current document's sort value (preceding="unbounded", following="current").

A document window frame (frame=DocumentWindowFrame(preceding, following)) lets the caller explicitly control how many neighboring documents before (preceding) and after (following) the current document are included in the window calculation (similar to SQL ROWS BETWEEN ... AND ...):

  • WindowBound.UNBOUNDED ("unbounded"): Extends to the first (preceding) or last (following) document in the partition.
  • WindowBound.CURRENT ("current"): Starts or ends at the current document.
  • Integer offset (int or Expression):
    • For preceding: a positive integer $N$ includes up to $N$ documents before the current document; a negative integer $-N$ starts the frame $N$ documents after the current document (look-ahead frame).
    • For following: a positive integer $N$ includes up to $N$ documents after the current document; a negative integer $-N$ ends the frame $N$ documents before the current document (look-behind frame excluding the current document).

For example, DocumentWindowFrame(WindowBound.UNBOUNDED, WindowBound.CURRENT) computes a running total and running count from the first book in each genre up to the current book:

title genre published rating running_rating (sum) running_count (Count)
"1984" "Dystopian" 1949 4.2 4.2 1
"The Handmaid's Tale" "Dystopian" 1985 4.1 8.3 2
"Dune" "Science Fiction" 1965 4.6 4.6 1
"The Hitchhiker's Guide to the Galaxy" "Science Fiction" 1979 4.2 8.8 2
from google.cloud.firestore_v1.pipeline_expressions import Count, Field
from google.cloud.firestore_v1.pipeline_types import (
    DocumentWindowFrame,
    WindowBound,
    WindowSpec,
)

pipeline = (
    client.pipeline()
    .collection("books")
    .add_window_fields(
        Field.of("rating").sum().as_("running_rating"),
        Count().as_("running_count"),
        window=WindowSpec(
            partition="genre",
            sort=Field.of("published").ascending(),
            frame=DocumentWindowFrame(
                WindowBound.UNBOUNDED, WindowBound.CURRENT
            ),
        ),
    )
)

Summary of Changes

What We Will Follow Up On

  • Value & Time-Range Window Frames (RangeWindowFrame): Follow-up PR adding RangeWindowFrame(preceding, following, unit=None) and widening WindowSpec(frame=...) to accept DocumentWindowFrame | RangeWindowFrame.
  • Per-Function Frame Overrides (.over(frame)): Follow-up PR adding WindowFunction.over(frame) so individual accumulators can override the stage's frame.
  • Enabling Live System Tests in CI (firestore: enable live window.yaml pipeline system tests once add_window_fields is in production #19115): Removing _skip_window_execution_tests in tests/system/test_pipeline_acceptance.py once the add_window_fields backend rollout is live in production.

Verification

  • Unit Tests: pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py (469 passed, 100% statement/branch coverage on pipeline_types.py).
  • System Tests: pytest tests/system/test_pipeline_acceptance.py -k "window.yaml" (verified both proto serialization and live sync/async execution against test-firestore.sandbox.googleapis.com:443: 6 passed).
  • Linter & Typecheck: mypy, ruff check --select I, ruff format --check, and flake8 google tests all pass cleanly.

… windows

Introduce WindowBound, _WindowFrame, and DocumentWindowFrame in
pipeline_types.py and add the documents parameter to WindowSpec to
support document-count sliding window frames (e.g., running totals and
moving averages) in AddWindowFields stages.

Export DocumentWindowFrame and WindowBound from google.cloud.firestore
and google.cloud.firestore_v1, and add unit and window.yaml system
tests.
@ohmayr
ohmayr requested review from a team as code owners November 18, 2026 01:44
Comment on lines +355 to +381
@dataclass
class DocumentWindowFrame(_WindowFrame):
"""Defines a document-count based window frame relative to the current document.

Example:
>>> from google.cloud.firestore_v1.pipeline_types import (
... DocumentWindowFrame,
... WindowBound,
... WindowSpec,
... )
>>> window = WindowSpec(
... documents=DocumentWindowFrame(
... WindowBound.UNBOUNDED, WindowBound.CURRENT
... )
... )

Args:
preceding: The lower bound of the frame (an integer offset, `WindowBound`, or `Expression`).
A positive integer includes up to that many documents before the current document.
A negative integer indicates a boundary following the current document.
following: The upper bound of the frame (an integer offset, `WindowBound`, or `Expression`).
A positive integer includes up to that many documents after the current document.
A negative integer indicates a boundary preceding the current document.
"""

preceding: WindowBound | str | int | pipeline_expressions.Expression
following: WindowBound | str | int | pipeline_expressions.Expression

@ohmayr ohmayr Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Note — DocumentWindowFrame Dataclass & _to_dict():

  • Chosen Design: DocumentWindowFrame(preceding, following) is a standalone @dataclass (inheriting _to_pb() from _WindowFrame and exposing _to_dict() -> {"documents": self._to_pb()}) passed to WindowSpec(frame=...), with symbolic bounds represented by WindowBound(str, Enum) (matching TimeUnit/PipelineDataType in this module).
  • Alternatives Considered & Why Rejected:
    • Builder method on WindowSpec (e.g., WindowSpec().documents(preceding, following) as in Java): Python pipeline option types (FindNearestOptions, SearchOptions, UnnestOptions) use keyword-argument constructors rather than fluent builders. A standalone frame class is also reusable directly in per-function .over(DocumentWindowFrame(...)) overrides in a follow-up PR.
    • Raw tuple or dict (frame=(-1, "current") or frame={"preceding": -1, "following": "current"}): A tuple hides which bound is preceding vs following and doesn't extend cleanly when RangeWindowFrame adds unit=...; a dict loses type annotations and autocomplete.

@ohmayr
ohmayr added this pull request to stack #19118 November 18, 2026 01:46

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for document-count based window frames in Firestore pipeline queries by adding WindowBound and DocumentWindowFrame classes, and integrating them into WindowSpec. The review feedback correctly identifies a critical bug where the __eq__ method of WindowSpec was not updated to include the new documents attribute, which would lead to incorrect equality evaluations. A code suggestion is provided to resolve this issue.

Replace WindowSpec(documents=...) with WindowSpec(frame=...) so nominal
window frame types use a single unified parameter without repeating the
frame kind. Add _to_dict() on DocumentWindowFrame to emit its wire map
entry directly.
| str
] = None,
sort: Optional[Sequence[Ordering] | Ordering] = None,
frame: Optional[DocumentWindowFrame] = None,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design Note — WindowSpec(frame=...) & Backend-Enforced sort Validation:

  • Chosen Design:
    1. WindowSpec accepts a single frame= parameter (widened to DocumentWindowFrame | RangeWindowFrame in the next PR) and delegates wire-key dispatch via self.frame._to_dict().
    2. WindowSpec.__init__ does not raise a client-side ValueError when frame is provided without sort.
  • Alternatives Considered & Why Rejected:
    • Separate documents= and range= keyword arguments (as in Node's plain-object OneOf): In Python, the frame argument is already a nominally typed class (DocumentWindowFrame vs RangeWindowFrame), so repeating the frame kind in the keyword name (documents=DocumentWindowFrame(...), range=RangeWindowFrame(...)) stutters, shadows built-in range, and requires runtime mutual-exclusivity validation (if documents is not None and range is not None: raise ValueError(...)). A single frame= parameter makes passing two frames syntactically impossible and matches WindowFunction.over(frame).
    • Raising ValueError in WindowSpec.__init__ when frame is set and sort is omitted: Rejected per the cross-SDK Window Functions spec—sort requirement checks are intentionally enforced by the backend at execution time rather than in WindowSpec.__init__ (which also avoids duplicating backend validation rules when per-function .over(frame) overrides are used).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable to me, but the fact that all the other libraries use a different pattern makes me a bit nervous. WindowSpec(documents=DocumentWindowFrame(1, "current")) seems more consistent with others, and I don't think a mutual exclusion check is too much of a burden. Maybe types could be simplified more if you wanted to avoid the naming stutter

But that said, I can't think of any real concerns with this current approach, and it does seem cleaner/nicer to me in isolation. So LGTM if you want to keep it as-is

>>> window = WindowSpec(
... sort=Field.of("published").ascending(),
... frame=DocumentWindowFrame(
... WindowBound.UNBOUNDED, WindowBound.CURRENT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the example could be more clear with preceding and following kwargs present

"partition=[Field.of('genre')], "
"sort=[Field.of('rating').descending()]"
"sort=[Field.of('rating').descending()], "
"frame=DocumentWindowFrame(preceding=-1, following=<WindowBound.CURRENT: 'current'>)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we fix the enum repr for <WindowBound.CURRENT: 'current'>?

spec = WindowSpec(
partition="genre",
sort=Field.of("rating").descending(),
frame=DocumentWindowFrame(-1, WindowBound.CURRENT),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't -1 mean "a boundary following the current document" here? So isn't this boundary backwards, where preceding is after following?

Args:
partition: Field paths or expressions used to partition documents into independent windows.
sort: Orderings specifying how documents are sorted within each partition.
frame: A `DocumentWindowFrame` defining a document-count window frame.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if None?

@dataclass
class DocumentWindowFrame(_WindowFrame):
"""Defines a document-count based window frame relative to the current document.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit more context to include:

    Bounds count documents, not values. `WindowBound.CURRENT` refers only to the
    current document's position, so documents with the same sort value are not
    automatically included in the frame.

    `sort` on the enclosing `WindowSpec` is optional for a document frame. If it
    is omitted, documents are processed in the order they reach the stage.

frame:
DocumentWindowFrame:
preceding: unbounded
following: current

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have a test with a numeric offset? (Maybe ask gemini for more test cases too)

| str
] = None,
sort: Optional[Sequence[Ordering] | Ordering] = None,
frame: Optional[DocumentWindowFrame] = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable to me, but the fact that all the other libraries use a different pattern makes me a bit nervous. WindowSpec(documents=DocumentWindowFrame(1, "current")) seems more consistent with others, and I don't think a mutual exclusion check is too much of a burden. Maybe types could be simplified more if you wanted to avoid the naming stutter

But that said, I can't think of any real concerns with this current approach, and it does seem cleaner/nicer to me in isolation. So LGTM if you want to keep it as-is

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants