Skip to content

Support rate family aggregation functions in table model - #18334

Open
CoollZzz wants to merge 1 commit into
apache:masterfrom
CoollZzz:feat/rate-functions
Open

Support rate family aggregation functions in table model#18334
CoollZzz wants to merge 1 commit into
apache:masterfrom
CoollZzz:feat/rate-functions

Conversation

@CoollZzz

@CoollZzz CoollZzz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Description

Related issue: #17976

This PR adds four built-in aggregation functions to the IoTDB table model: rate(), increase(), irate(), and delta(). Their calculation semantics are based on the corresponding Prometheus range-vector functions, while window boundaries follow IoTDB's [window_start, window_end) convention.

Function semantics

Function Signature Description
rate rate(value, time, window_start, window_end) Calculates the extrapolated average per-second growth rate of a counter and compensates for counter resets.
increase increase(value, time, window_start, window_end) Calculates the extrapolated total increase of a counter and compensates for counter resets.
irate irate(value, time) Calculates the instantaneous per-second counter rate using the last two valid samples.
delta delta(value, time, window_start, window_end) Calculates the extrapolated difference of a gauge without counter-reset compensation.

The supported value types are INT32, INT64, FLOAT, and DOUBLE. Time and window arguments accept TIMESTAMP and INT64, with INT64 interpreted using the current timestamp_precision. All four functions return DOUBLE.

rate(), increase(), and delta() use all valid samples in the aggregation group and extrapolate the result to the window boundaries. Extrapolation uses the average sampling interval, a 1.1 threshold for distant boundaries, and counter zero-point protection where applicable. irate() only uses the last two valid samples and does not perform boundary extrapolation.

Accumulator and planner design

Both TableAccumulator and GroupedAccumulator execution paths are supported.

Each function has its own abstract accumulator class for shared function-specific validation, window handling, and result calculation. Ordered and naive accumulators are implemented as separate concrete classes because their state structures and lifecycle behavior differ significantly.

  • Ordered accumulators are selected only for the SINGLE aggregation step when the physical planner proves that the function's time argument is ascending within every SQL aggregation group. They consume samples incrementally without buffering and sorting all input rows.
  • Naive accumulators buffer samples and sort them by time before evaluation. They support unordered input and all multi-stage aggregation steps.
  • Multi-stage aggregation uses a versioned BLOB intermediate state containing the complete sample state and window boundaries.
  • Naive table and grouped accumulators report retained buffer memory through the existing memory reservation framework.
  • The four functions are not pushed into aggregation table scans because their explicit time argument may differ from the physical table time column and their calculations require raw samples.

The ordered-input property is propagated through AggregationNode serialization so that distributed execution selects the same accumulator implementation as the coordinator plan.

Validation and edge cases

The implementation handles the following cases:

  • NULL values are ignored.
  • Fewer than two valid samples return NULL.
  • Counter resets are compensated for rate(), increase(), and irate().
  • Negative values are rejected for counter functions but are supported by delta().
  • Non-finite values, including NaN and positive or negative infinity, are rejected.
  • Required time or window arguments cannot be NULL for a valid sample.
  • Window boundaries must be consistent within an aggregation group and satisfy window_start < window_end.
  • Every valid sample must satisfy window_start <= time < window_end.
  • Duplicate timestamps within the same aggregation group are rejected.
  • Non-finite intermediate or extrapolated results are rejected.

Tests

Integration tests were added to IoTDBTableAggregationIT for:

  • Normal global, grouped, ordered, unordered, distributed, and windowed aggregation.
  • Counter resets, multiple resets, zero-point protection, extrapolation thresholds, null values, missing samples, and gap-fill scenarios.
  • All supported numeric and time argument types.
  • Invalid argument counts and types, non-finite and negative counter values, invalid or inconsistent windows, out-of-window samples, null time arguments, and duplicate timestamps.

The four target test methods were verified with both the TableSimpleIT and TableClusterIT profiles. The full reactor test compilation also passed for both the default and Chinese locales.


This PR has:

  • been self-reviewed.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.calc.execution.operator.source.relational.aggregation.rate
  • org.apache.iotdb.calc.execution.operator.source.relational.aggregation.grouped.rate
  • AccumulatorFactory
  • TableMetadataImpl
  • TableDistributedPlanGenerator
  • AggregationNode
  • PushAggregationIntoTableScan
  • IoTDBTableAggregationIT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant