Skip to content

Pipe: Respect history enable with source time range - #18362

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/source-time-respect-history-enable
Open

Pipe: Respect history enable with source time range#18362
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/source-time-respect-history-enable

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Respect source.history.enable

A global source.start-time or source.end-time previously enabled historical extraction unconditionally. Consequently, source.history.enable=false could still transfer data that existed before pipe creation.

This PR:

  • determines historical enablement independently from the global time range;
  • preserves the default behavior when source.history.enable is omitted;
  • preserves historical recovery for pipe restarts and newly added DataRegions;
  • continues applying the global time range to realtime extraction and to historical extraction when history is enabled.

Keep parameter precedence clear

A global start/end time still takes precedence over history-specific start/end time. The validation warning is updated so that it no longer reports source.history.enable as invalid when a global time range is configured.

Tests

  • PipeHistoricalDataRegionTsFileAndDeletionSourceTest: 19 tests, 0 failures.
  • IoTDBPipeSourceIT#testSourceTimeRangeRespectsHistoryDisable: 1 test, 0 failures.
  • Spotless checks for the modified DataNode and integration-test code.
  • git diff --check.

This PR has:

  • been self-reviewed.
  • added comments explaining the why and intent of the code.
  • added unit tests covering the new code path.
  • added integration tests.
  • been tested in the integration-test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeHistoricalDataRegionTsFileAndDeletionSource
  • IoTDBDataRegionSource
  • PipeHistoricalDataRegionTsFileAndDeletionSourceTest
  • IoTDBPipeSourceIT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants