[DSIP-95][API] Complete the functionality of using dependencies in the complement data by det101 · Pull Request #18003 · apache/dolphinscheduler

det101 · 2026-02-27T08:17:24Z

Was this PR generated or assisted by AI?

yes. The UT portion is generated by AI, Main functional code AI assisted

Purpose of the pull request

Brief change log

The function of supplementing data supports dependency relationships: During the process of supplementing data, it can identify the dependency relationships of workflows and recursively pull up downstream dependent workflows

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Copilot

Pull request overview

Implements downstream workflow triggering for complement/backfill runs in the API layer, adding support for “trigger dependent workflows” behavior and accompanying unit tests.

Changes:

Implemented doBackfillDependentWorkflow to fetch downstream workflow definitions and trigger backfill runs for them.
Added visited-code tracking intended to prevent self/cyclic triggering and duplicate downstream triggers.
Added BackfillWorkflowExecutorDelegateTest with basic scenarios for downstream triggering and filtering.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java	Adds dependent workflow backfill triggering logic and wiring for lineage + workflow definition lookup.
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java	Adds unit tests for the new dependent backfill triggering logic (single-hop scenarios).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

SbloodyS

You didn't fill in the content according to the PR template, please fix it.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java:119

In parallel mode, expectedParallelismNumber can be 0 (the validator only rejects values < 0). If it is 0, splitDateTime(listDate, expectedParallelismNumber) will divide by zero and throw ArithmeticException. Treat 0 the same as null (default to listDate.size()), or explicitly guard against <= 0 before calling splitDateTime.

        final BackfillWorkflowDTO.BackfillParamsDTO backfillParams = backfillWorkflowDTO.getBackfillParams();
        Integer expectedParallelismNumber = backfillParams.getExpectedParallelismNumber();

        List<ZonedDateTime> listDate = backfillParams.getBackfillDateList();
        if (expectedParallelismNumber != null) {
            expectedParallelismNumber = Math.min(listDate.size(), expectedParallelismNumber);
        } else {
            expectedParallelismNumber = listDate.size();
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

Copilot · 2026-03-25T08:56:54Z

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

+    public void testDoParallelBackfillWorkflow_ShouldIsolateVisitedCodesAcrossChunks() {
+        long upstreamCode = 500L;
+        WorkflowDefinition upstreamWorkflow =
+                WorkflowDefinition.builder().code(upstreamCode).releaseState(ReleaseState.ONLINE).build();
+        List<ZonedDateTime> dates = Arrays.asList(
+                ZonedDateTime.parse("2026-02-01T00:00:00Z"),
+                ZonedDateTime.parse("2026-02-02T00:00:00Z"),
+                ZonedDateTime.parse("2026-02-03T00:00:00Z"));
+        BackfillWorkflowDTO.BackfillParamsDTO params = BackfillWorkflowDTO.BackfillParamsDTO.builder()
+                .runMode(RunMode.RUN_MODE_PARALLEL)
+                .backfillDateList(dates)
+                .expectedParallelismNumber(2)
+                .backfillDependentMode(ComplementDependentMode.ALL_DEPENDENT)
+                .allLevelDependent(true)
+                .executionOrder(ExecutionOrder.ASC_ORDER)
+                .build();
+        BackfillWorkflowDTO dto = BackfillWorkflowDTO.builder()
+                .workflowDefinition(upstreamWorkflow)
+                .backfillParams(params)
+                .build();
+        Set<Long> baseVisitedCodes = new HashSet<>(Collections.singleton(upstreamCode));
+        List<Set<Long>> visitedSnapshotPerChunk = new java.util.ArrayList<>();
+
+        doAnswer(invocation -> {
+            Set<Long> chunkVisited = invocation.getArgument(2);
+            visitedSnapshotPerChunk.add(new HashSet<>(chunkVisited));
+            chunkVisited.add(9000L + visitedSnapshotPerChunk.size());
+            return null;
+        }).when(backfillWorkflowExecutorDelegate).doBackfillDependentWorkflowForTesting(any(), any(), any());
+
+        List<Integer> result = backfillWorkflowExecutorDelegate.executeWithVisitedCodes(dto, baseVisitedCodes);
+


This test calls executeWithVisitedCodes, which will run the real doBackfillWorkflow and attempt to use registryClient / Clients.withService(IWorkflowControlClient) to contact a master. Since neither is mocked/stubbed in this test, it will fail with NPE or a ServiceException before exercising the visited-codes isolation assertions. Consider refactoring to unit-test the chunk visited-code cloning without invoking the master trigger, or add a test seam/mocking for the backfill trigger step.

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

SbloodyS

Please fill in the content according to the PR description template. Many parts of your code are written by ai, but there is no statement in PR.

det101 · 2026-04-03T02:50:37Z

Please fill in the content according to the PR description template. Many parts of your code are written by ai, but there is no statement in PR.

done

SbloodyS

LGTM

det101 · 2026-04-07T07:47:07Z

@caishunfeng Please help review

ruanwenjun

It's better to split into two methods.
First, find out all downstream workflows (This method might be useful at other api).
Second, trigger them.

det101 · 2026-04-08T01:30:18Z

It's better to split into two methods. First, find out all downstream workflows (This method might be useful at other api). Second, trigger them.

@ruanwenjun Could you confirm whether “all downstream” here means only immediate dependents (same as today, just refactored), or all reachable downstream workflows (transitive closure)? I can go either way once I know which you want.

Split downstream discovery from execution, add transitive resolution coverage, and keep parallel backfill triggering downstream dates by isolating visited state per chunk. Made-with: Cursor

det101 · 2026-04-08T10:53:52Z

It's better to split into two methods. First, find out all downstream workflows (This method might be useful at other api). Second, trigger them.

I split the backfill dependent logic into resolve downstream + trigger downstream, and fixed the parallel case so downstream workflows no longer miss dates across chunks. @ruanwenjun

…ing for backfill expansion

ruanwenjun

Basically LGTM

ruanwenjun · 2026-04-14T08:27:53Z

...i/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowLineageServiceImpl.java

+                                                                             boolean filterOfflineWorkflow) {
+
+        Set<Long> resultCodes = new LinkedHashSet<>();
+        Set<Long> expandedCodes = new HashSet<>();


Suggested change

Set<Long> expandedCodes = new HashSet<>();

Set<Long> visitedWorkflowCodes = new HashSet<>();

…18003

sonarqubecloud · 2026-04-14T09:16:11Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

det101 requested review from SbloodyS and caishunfeng as code owners February 27, 2026 08:17

det101 force-pushed the DSIP-95 branch from 91a1353 to a3fe99a Compare February 27, 2026 08:18

SbloodyS requested a review from Copilot March 20, 2026 03:50

Copilot started reviewing on behalf of SbloodyS March 20, 2026 03:51 View session

SbloodyS assigned det101 Mar 20, 2026

SbloodyS added backend DSIP labels Mar 20, 2026

SbloodyS added this to the 3.4.2 milestone Mar 20, 2026

github-actions bot added the test label Mar 20, 2026

Copilot AI reviewed Mar 20, 2026

View reviewed changes

SbloodyS reviewed Mar 20, 2026

View reviewed changes

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Outdated Show resolved Hide resolved

SbloodyS closed this Mar 23, 2026

SbloodyS reopened this Mar 23, 2026

det101 requested review from SbloodyS and Copilot March 24, 2026 03:20

Copilot started reviewing on behalf of det101 March 24, 2026 03:21 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

det101 force-pushed the DSIP-95 branch 2 times, most recently from ca11eb6 to 1841eaf Compare March 25, 2026 08:51

det101 requested a review from Copilot March 25, 2026 08:51

Copilot started reviewing on behalf of det101 March 25, 2026 08:52 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

det101 requested a review from Copilot March 25, 2026 10:02

Copilot started reviewing on behalf of det101 March 25, 2026 10:02 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Outdated Show resolved Hide resolved

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Show resolved Hide resolved

This was referenced Mar 26, 2026

[WIP] Evaluate accuracy of code review conclusions det101/det101#1

Closed

[WIP] Fix timezone round-trip bug in BackfillWorkflowExecutorDelegate det101/dolphinscheduler#3

Closed

det101 requested a review from Copilot March 26, 2026 01:40

Copilot started reviewing on behalf of det101 March 26, 2026 01:41 View session

luxl added 3 commits March 31, 2026 09:11

fix spotless

1a53393

fix spotless

37ddf8a

Fix JDK version syntax compatibility issues

a8aac0d

ruanwenjun force-pushed the DSIP-95 branch from 57b94ac to a8aac0d Compare March 31, 2026 01:11

Merge branch 'dev' into DSIP-95

fc9fd1e

SbloodyS requested changes Mar 31, 2026

View reviewed changes

SbloodyS and others added 2 commits April 7, 2026 13:53

Merge branch 'dev' into DSIP-95

585fc36

Merge dev into DSIP-95

364e7bc

SbloodyS closed this Apr 7, 2026

SbloodyS reopened this Apr 7, 2026

SbloodyS previously approved these changes Apr 7, 2026

View reviewed changes

SbloodyS requested a review from ruanwenjun April 7, 2026 07:48

ruanwenjun reviewed Apr 7, 2026

View reviewed changes

Refactor dependent backfill into resolve and trigger methods.

725f180

Split downstream discovery from execution, add transitive resolution coverage, and keep parallel backfill triggering downstream dates by isolating visited state per chunk. Made-with: Cursor

det101 dismissed SbloodyS’s stale review via 725f180 April 8, 2026 10:48

det101 requested a review from ruanwenjun April 9, 2026 06:30

det101 and others added 3 commits April 9, 2026 15:24

Merge branch 'dev' into DSIP-95

fb41680

Implement downstream workflow resolution with optional offline filter…

308a6c2

…ing for backfill expansion

Merge branch 'dev' into DSIP-95

3ab0433

ruanwenjun reviewed Apr 14, 2026

View reviewed changes

luxl and others added 3 commits April 14, 2026 16:50

fix spotless

5bc37c5

Merge branch 'DSIP-95' of github.com:det101/dolphinscheduler into pr-…

0fa16ce

…18003

Merge branch 'dev' into DSIP-95

57f2ef9

ruanwenjun closed this Apr 14, 2026

ruanwenjun reopened this Apr 14, 2026

	Set<Long> expandedCodes = new HashSet<>();
	Set<Long> visitedWorkflowCodes = new HashSet<>();

Conversation

det101 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was this PR generated or assisted by AI?

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

det101 commented Apr 3, 2026

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

det101 commented Apr 7, 2026

Uh oh!

ruanwenjun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

det101 commented Apr 8, 2026

Uh oh!

det101 commented Apr 8, 2026

Uh oh!

ruanwenjun left a comment

Choose a reason for hiding this comment

Uh oh!

ruanwenjun Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 14, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

det101 commented Feb 27, 2026 •

edited

Loading

ruanwenjun left a comment •

edited

Loading