Skip to content

[IT] Reduce integration test runtime - #19242

Merged
jt2594838 merged 6 commits into
apache:masterfrom
jt2594838:codex/optimize-it-runtime
Nov 13, 2026
Merged

jt2594838 merged 6 commits into
apache:masterfrom
jt2594838:codex/optimize-it-runtime

Conversation

@jt2594838

@jt2594838 jt2594838 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Compatible integration-test methods repeatedly start identical clusters, create clients for individual writes/flushes, and wait through fixed polling or observation delays. This change shares class-owned clusters, isolates or resets scenario data and metadata, reuses Sessions, and batches fixture writes. The DataSink fixture reduces repeated Tablet writes from 125 to 12 per generator while retaining four write/flush rounds, both databases and all five table shapes. Query fixtures retain their input values; recovery cases retain their required restarts.

Configure the MQTT payload fixtures with a single broker callback worker so CONNECT registers its session before each single-publish input is processed. Use immediate, bounded condition polling for readiness, quota/CQ/TTL visibility and asynchronous cleanup. Data-only Pipe cases wait for a full-flush completion barrier and then check the exact receiver results; negative permission/filter observations retain their required observation windows. Completion waits select only Regions admitted by the production tree/table database filters, while requiring every selected task and live gauge. CQ polling checks each DataNode's row count independently within its existing deadline before the full final result comparison through the cluster wrapper; SQL and transport errors still propagate. Restore dynamic configuration between shared-cluster methods, bound the LOAD fixture's Region budget, and make acknowledged-write durability explicit with SYNC WAL in restart fixtures. Shared-cluster cleanup tolerates only an already-deleted database or pipe when removal finishes after its metadata snapshot; other cleanup failures still fail the test. Add opt-in ConfigNode startup tracing and client-side phase timings to explain remaining costs. All changes are in integration-test.

Measured once per version, serially on the same Windows machine at 592d6635027c0a4dea0fa461f2bb6f0c56bc15e8, before porting to current master. Class durations include fixture setup/cleanup. Of 165 class/profile combinations, 156 passed on both versions with identical method/skip sets and valid execution evidence; nine with baseline failures are excluded from the savings:

Comparable tests Before (s) After (s) Reduction
Pipe IT 8,826.987 2,551.928 71.1%
Other IT 7,576.007 2,505.097 66.9%
Total 16,402.994 5,057.025 69.2%

The optimized original-revision run recorded 1,417 passed and 73 skipped with zero failures/errors. These are single-pair local measurements, not CI timing estimates. The final patch preserves upstream master regressions and adds the target branch's credential/namespace adaptations; current-master validation is reported separately below.

Validation:

  • Spotless, IT discovery/shard coverage, patch application to frozen master, new-file license headers, and en/zh message-key/placeholder parity checked.

  • Default and Chinese full-reactor test compilation passed (51 modules per locale).

  • On 7b407ec00292: 165 class/profile combinations, 1454 passed, 71 skipped, 0 failures and 0 errors. The complete first run exposed fixture isolation/durability issues; after fixing them, affected classes and descendants of the repaired shared fixture were rerun in full. Method/category coverage was checked against the source, with no new Ignore annotations.

  • Follow-up CI regressions: the filtered Table Source fixture reproduces the missing-completion-metric timeout with the original helper; changing only the helper passes the same 1C3D input. The Table Source class passes on 1C1D and 1C3D, the two Tree Source completion cases pass on both topologies, and all five CQ cases at millisecond/microsecond/nanosecond precision pass under SimpleIT and ClusterIT. Total: 10 class/profile combinations, 42 passed, 8 existing skips, no failures/errors. These are functional follow-up runs, not an additional before/after runtime measurement.

  • MQTT callback follow-up: both complete MQTT payload classes pass under their single- and three-DataNode profiles (4 class/profile combinations, 18 passed, no skips/failures/errors), with the effective one-worker configuration checked in all eight DataNode logs. A local probe of the actual Moquette executor deterministically shows PUBLISH overtaking a held CONNECT with two workers and ordered execution with one; this probe does not run an IoTDB server. The previous JSON class also passed a local rerun, so that rerun is not claimed as an end-to-end reproduction of the Windows CI timeout.

  • Tree Pipe cleanup follow-up: all five classes selected by the failed basic shard pass on Windows under MultiClusterIT2DualTreeAutoBasic with HighPerformanceMode on both environments (32 passed, no skips/failures/errors). The actual cleanup helper is checked against six controlled RPC cases per version: the previous helper rejects PIPE_NOT_EXIST_ERROR after a successful SHOW, while the repair accepts it; unrelated status and transport errors still fail, and system pipes remain untouched. This controlled probe does not start an IoTDB server.

  • Master merge follow-up: merge cca1dbbde6d0, retaining all upstream CQ permission and Session constant-expression regressions. Resolve the CQ import conflict by keeping both dependencies, restore the resampling comment to its original test, and describe the four permission wrappers. Fresh matching node templates and full-reactor en/zh test compilation pass. 10 complete class/profile combinations pass (108 passed, 0 skipped, no failures/errors), covering CQ execution at ms/us/ns, CQ validation, and Table Session on single- and three-DataNode profiles.
    Master advanced to 9a281f8e1c06 during validation. Its snapshot-recovery change has no overlap with this PR; Git merge-tree confirms the tested candidate merges cleanly with that revision. The later upstream commit is outside the runtime validation scope above.

  • Table-model Pipe snapshot cleanup follow-up: both shared Table cleanup dialects and the Tree AutoDrop class's Table cleanup accept only PIPE_NOT_EXIST_ERROR after SHOW PIPE, while keeping every other failure visible. The two complete AutoDrop classes pass on Windows with their MultiClusterIT2DualTableManualEnhanced and MultiClusterIT2DualTreeAutoEnhanced profiles and HighPerformanceMode on both environments (6 passed, no skips/failures/errors). Both actual helpers pass 16 controlled RPC checks per version; the previous code rejects the missing-pipe snapshot case and the new code accepts it and continues cleanup. The controlled probe does not start an IoTDB server.

CI runtime comparison (2026-11-13)

Measured CI revision: 44825a9e34ee6dc9282d7754a5c188e27dfa91c3. The ordinary Tree/Table IT, Pipe IT and Edge IT workflows passed on their first attempts.

The primary controls are the same-day runs of #17871 (a8e5d9e1f70c, 04:42 UTC), #19224 (2608feb6b1f6, 07:03 UTC), and #19247 (665c926a36d7, 03:41 UTC), compared with this revision scheduled at 06:51 UTC. Their workflow YAML, sharding script, job matrices and Failsafe execution settings match. The target IT POM adds only the locale-source build-helper mapping; #19224/#19247 also use an older jline dependency. This is an observational CI comparison across PRs, with one run per version, rather than a controlled before/after experiment.

Cumulative test-step duration, including Maven build time. Sum all matching jobs within each control workflow, then take the median of those complete-workflow totals. Queue, checkout, cache and upload steps are excluded. The totals below are accumulated parallel-job minutes, not elapsed workflow time.

Workflow Jobs Successful controls Control median (min) This PR (min) Change
Tree 1C1D 4 3 141.05 114.40 18.9% less
Tree 1C3D 1 2 41.45 41.32 0.3% less
Pipe 17 3 333.78 193.78 41.9% less
Table 1C1D 4 3 131.33 105.60 19.6% less
Table 1C3D 1 3 46.40 35.25 24.0% less
Edge 2 4 4.63 5.03 8.6% more

The Pipe workflow's slowest test step decreased from a control median of 36.47 min to 28.97 min; Tree 1C1D decreased from 52.37 to 39.48 min and Table 1C1D from 47.33 to 31.63 min. Tree 1C3D was essentially unchanged. Edge was slower than its control median, within the observed control range.

IT execution phase only. Log timestamps from the integration-test module's Failsafe start to its final test summary exclude compilation and earlier dependency-module tests; they include fork/cluster startup and cleanup. These are also sums across matching jobs.

Workflow Control median (min) This PR (min) Change
Tree 1C1D 120.04 92.57 22.9% less
Tree 1C3D 35.00 34.39 1.8% less
Table 1C1D 104.74 79.64 24.0% less
Table 1C3D 38.38 28.63 25.4% less
Edge (Ubuntu Java IT) 0.40 0.41 3.6% more

Pipe success jobs redirect Maven output to a local file that is not uploaded on success, so a pure Failsafe or class-level CI duration cannot be recovered for Pipe. The available execution logs show no failed-test output, transfer retries, or masked-success branch for the selected runs.

Coverage and interpretation:

Control workflow runs used for the timing medians
Workflow Control runs
Tree 1C1D #17871, #19224, #19247
Tree 1C3D #17871, #19224
Pipe #17871, #19224, #19247
Table 1C1D #17871, #19224, #19247
Table 1C3D #17871, #19224, #19247
Edge #17871, #19224, #19247, #19258

Status checked at 2026-11-13T08:56:12.430409+00:00: Unit-Test is in_progress; AINode 1C1D1A is queued. Incomplete workflows are excluded from these timing results.

- Report the tested revision, edition, profile, and result accurately. Passing an upstream build
does not establish that the corresponding downstream edition was tested.

### Startup diagnostics

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document the readiness time budgets and opt-in startup tracing so future test changes preserve the retry deadline and interpret nested JVM timings correctly.

Comment thread integration-test/pom.xml
</dependencies>
<build>
<plugins>
<!-- Compile readiness diagnostics in the selected locale, using the parent source mapping. -->

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enable the inherited locale source mapping for the new readiness diagnostics. Both en and zh source sets were compiled in the full reactor; no test profile or category selection changes here.

package org.apache.iotdb.it.env.cluster.env;

/** Localized diagnostics for the integration-test readiness deadlines. */
public final class IntegrationTestEnvMessages {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep readiness-deadline failure messages in the selected compile-time locale. The English and Chinese classes have the same three keys and format specifiers, preserving endpoint and failure diagnostics.

/** Localized diagnostics for the integration-test readiness deadlines. */
public final class IntegrationTestEnvMessages {

private IntegrationTestEnvMessages() {}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep readiness-deadline failure messages in the selected compile-time locale. The English and Chinese classes have the same three keys and format specifiers, preserving endpoint and failure diagnostics.

package org.apache.iotdb.it.env.cluster.env;

/** Localized diagnostics for the integration-test readiness deadlines. */
public final class IntegrationTestEnvMessages {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep readiness-deadline failure messages in the selected compile-time locale. The English and Chinese classes have the same three keys and format specifiers, preserving endpoint and failure diagnostics.

import org.apache.iotdb.itbase.category.ClusterIT;
import org.apache.iotdb.rpc.TSStatusCode;

import org.awaitility.Awaitility;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// Partition allocation excludes Removing and ReadOnly nodes, permits an offline node, and

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Partition allocation excludes Removing and ReadOnly nodes, permits an offline node, and restores all regions after that node restarts.

Poll immediately and then every 100 ms for the observable condition while keeping a bounded wait and preserving the scenario assertion.

import org.apache.iotdb.rpc.TSStatusCode;

import org.junit.After;
import org.awaitility.Awaitility;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.


@Before
public void setUp() throws Exception {
// All scenarios use isolated database names so one initialized 1C1D cluster can serve the class.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.


@After
public void tearDown() {
@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// Expired partitions are eventually removed for database and table TTL rules.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Expired partitions are eventually removed for database and table TTL rules.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

Assert.fail("The PartitionTable in the ConfigNode is not auto cleaned!");
}

// Device-level TTL alone must not clean expired partitions when the database has no TTL.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Device-level TTL alone must not clean expired partitions when the database has no TTL.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// Expired partitions are eventually removed when a table-model database has a TTL.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Expired partitions are eventually removed when a table-model database has a TTL.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Assert.fail("The PartitionTable in the ConfigNode is not auto cleaned!");
}

private static void awaitPartitionTableAutoCleaned(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poll immediately and then every 100 ms for the observable condition while keeping a bounded wait and preserving the scenario assertion.

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.awaitility.Awaitility;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

statement.execute(String.format("delete database %s", SchemaConfig.DATABASE_0));
statement.execute(String.format("delete database %s", SchemaConfig.DATABASE_1));
} catch (final IoTDBSQLException ignored) {
deleteDatabaseIfExists(statement, SchemaConfig.DATABASE_0);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this LOAD fixture's databases plus its explicitly tracked template and user. Templates and users outlive database deletion and would otherwise contaminate later methods sharing the nodes.

}
}

private void deleteDatabaseIfExists(final Statement statement, final String database)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check only the fixture-owned database before deletion so empty/partially initialized LOAD scenarios can be cleaned without a global database-count assumption.


private boolean deleteDir() {
for (final File file : tmpDir.listFiles()) {
if (tmpDir == null) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep temporary LOAD files owned by the method, including partially initialized setup. Delete nested generated files where applicable so cluster reuse does not retain the previous input files.

return tmpDir.delete();
}

private boolean deleteDirectoryContents(final File directory) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove nested LOAD input directories created by the upstream regression cases before deleting their parent temporary directory. Keep a failed deletion observable to the caller.

}
}

// Loading aligned and non-aligned devices extends the bound template and preserves every point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Loading aligned and non-aligned devices extends the bound template and preserves every point.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// LOAD must reject missing SYSTEM/schema privileges and succeed after both grants are present.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: LOAD must reject missing SYSTEM/schema privileges and succeed after both grants are present.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

import org.apache.iotdb.itbase.category.LocalStandaloneIT;

import org.junit.After;
import org.junit.AfterClass;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.


@Before
public void setUp() throws Exception {
@BeforeClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize the class's fixed node configuration once for compatible methods. Per-method data/state preparation or isolated namespaces preserve the individual scenarios without repeated cluster startup.

EnvFactory.getEnv().initClusterEnvironment();
}

@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

}

@Before
public void setUp() throws SQLException {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

import org.apache.tsfile.write.record.Tablet;
import org.apache.tsfile.write.schema.MeasurementSchema;
import org.junit.After;
import org.junit.AfterClass;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.


@Before
public void setUp() throws Exception {
@BeforeClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize the class's fixed node configuration once for compatible methods. Per-method data/state preparation or isolated namespaces preserve the individual scenarios without repeated cluster startup.

EnvFactory.getEnv().initClusterEnvironment();
}

@Before

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.


@After
public void tearDown() throws Exception {
cleanTestDatabases();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this method's test-owned data/schema after execution while retaining class-owned nodes. The next case prepares its own input and cannot inherit the previous result.

cleanTestDatabases();
}

@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

EnvFactory.getEnv().cleanClusterEnvironment();
}

private void cleanTestDatabases() throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

// Two-device writes cross a partition boundary; overwrites retain values and flush releases

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Two-device writes cross a partition boundary; overwrites retain values and flush releases memory.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.


package org.apache.iotdb.db.it.aggregation;

import org.apache.iotdb.isession.ISession;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

private static void prepareData() {
try (Connection connection = EnvFactory.getEnv().getConnection();
Statement statement = connection.createStatement()) {
// Reuse one session for schema creation, typed batch writes and flushes. These tests exercise

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

private static void insertRange(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

// The row count signals that asynchronous CQ writes are ready without a fixed post-boundary

@jt2594838 jt2594838 Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poll immediately and then every 100 ms for the complete CQ output. CQ writes can fall between the individual DataNode query snapshots, so count each node's rows independently and wait until every count is sufficient within the unchanged 30-second deadline. SQL/transport errors still propagate; no exception is ignored. The caller still uses the cluster wrapper to compare every final timestamp and value. Apply the same condition handling at microsecond and nanosecond precision.

import org.apache.iotdb.itbase.category.ClusterIT;
import org.apache.iotdb.itbase.category.LocalStandaloneIT;

import org.awaitility.Awaitility;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// With microsecond timestamps, a 2s resampling CQ must write the expected maximum for every 1s

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With microsecond timestamps, a 2s resampling CQ must write the expected maximum for every 1s window after its future boundary.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// With microsecond timestamps, a CQ without EVERY and with a 4s range must produce the five

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With microsecond timestamps, a CQ without EVERY and with a 4s range must produce the five expected 1s-window maxima.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// With microsecond timestamps, a 4s-range CQ must fill the empty 1s window with 100 and write

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With microsecond timestamps, a 4s-range CQ must fill the empty 1s window with 100 and write the expected maxima for the populated windows.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// With microsecond timestamps, a CQ with RANGE 2s, 1s must write only the two expected 1s-window

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With microsecond timestamps, a CQ with RANGE 2s, 1s must write only the two expected 1s-window maxima at its 2s resampling boundaries.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// With microsecond timestamps, an align-by-device CQ must add one to all ten input values

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With microsecond timestamps, an align-by-device CQ must add one to all ten input values while preserving their half-second timestamps across overlapping 4s ranges.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

}
}

// The row count signals that asynchronous CQ writes are ready without a fixed post-boundary

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poll immediately and then every 100 ms for the complete CQ output. CQ writes can fall between the individual DataNode query snapshots, so count each node's rows independently and wait until every count is sufficient within the unchanged 30-second deadline. SQL/transport errors still propagate; no exception is ignored. The caller still uses the cluster wrapper to compare every final timestamp and value. Apply the same condition handling at microsecond and nanosecond precision.

import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.UUID;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.


@Before
public void setUp() throws Exception {
DataNodeWrapper dataNodeWrapper = EnvFactory.getEnv().getDataNodeWrapper(0);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

i * 1000,
x + Math.random() * y % (y - x + 1),
x + Math.random() * y % (y - x + 1))));
// Queries run after setup, so batch each device's rows to reduce SQL requests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bounded INSERT groups while preserving every generated point, deletion and flush boundary; this removes per-row RPC overhead from the large deletion regression.

@Category({LocalStandaloneIT.class})
public class DQualityIT {
protected static final int ITERATION_TIMES = 10_000;
private static final int INSERT_BATCH_SIZE = 100;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bounded INSERT groups while preserving every generated point, deletion and flush boundary; this removes per-row RPC overhead from the large deletion regression.

(int) Math.floor(a + Math.random() * b % (b - a + 1)),
x + Math.random() * y % (y - x + 1),
x + Math.random() * y % (y - x + 1))));
for (int start = 1; start <= ITERATION_TIMES; start += INSERT_BATCH_SIZE) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bounded INSERT groups while preserving every generated point, deletion and flush boundary; this removes per-row RPC overhead from the large deletion regression.

@Category({LocalStandaloneIT.class})
public class FrequencyIT {
protected static final int ITERATION_TIMES = 16384;
private static final int INSERT_BATCH_SIZE = 100;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bounded INSERT groups while preserving every generated point, deletion and flush boundary; this removes per-row RPC overhead from the large deletion regression.

Math.sin(i % PERIOD_1 * 2 * pi / (double) PERIOD_1),
Math.sin(i % PEROID_2 * 2 * pi / (double) PEROID_2),
0.5 * Math.sin(i % PERIOD_1 * pi) + Math.sin(i % PEROID_2 * pi)));
// Batch the same samples to reduce SQL requests, retaining the six-decimal input rounding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use bounded INSERT groups while preserving every generated point, deletion and flush boundary; this removes per-row RPC overhead from the large deletion regression.


import static org.awaitility.Awaitility.await;

public class PipeTestUtils {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use live JMX completion/replication state and metadata-derived expected task owners. Missing tasks, errors, changed pipe incarnations or leader changes must not look complete; callers still check the exact receiver result after the barrier.


private static final long COMPLETION_REFLUSH_INTERVAL_MILLIS = 1000;

private PipeTestUtils() {}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use live JMX completion/replication state and metadata-derived expected task owners. Missing tasks, errors, changed pipe incarnations or leader changes must not look complete; callers still check the exact receiver result after the barrier.

private final BaseEnv[] environments;
private final ISession[] sessions = new ISession[2];

@TestOnly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lazily retain one Session per environment across failed assertion polls. The caller owns this handler and closes it before cluster reset/restart; a successful first poll needs no retry clients.

environments = new BaseEnv[] {firstEnv, secondEnv};
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flush both environments through lazily retained Sessions so 100 ms assertion retries do not repeatedly create multi-node JDBC connections. Propagate write/flush failures with their cause.

}
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close both retry Sessions even when closing the first fails. Keep ownership at the scenario boundary so clients cannot leak into a reset or restarted cluster.

}
}

// ALTER must reject a missing table-model pipe, and the same connection must then create it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: ALTER must reject a missing table-model pipe, and the same connection must then create it.

Reuse the method-owned JDBC statement across the invalid CREATE/ALTER combinations. Keep each expected rejection and the pipe metadata checks, while class cleanup removes test pipes before the next shared-cluster method.

@RunWith(IoTDBTestRunner.class)
@Category({MultiClusterIT2DualTableManualBasic.class})
public class IoTDBPipeDataSinkIT extends AbstractPipeTableModelDualManualIT {
public class IoTDBPipeDataSinkIT extends AbstractPipeTableModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

// overlapping writes across flushes, with 12 writes instead of 125 (approximately one tenth).
private static final int[] TABLETS_PER_ROUND = {5, 5, 1, 1};

@Rule public final TestName testName = new TestName();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

super.setUp();
}

// With realtime-first disabled, batched thrift transfer must deliver both history/realtime

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: With realtime-first disabled, batched thrift transfer must deliver both history/realtime table rows and tree points.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

// Tablet-format transfer must preserve the mixed history/realtime data in both models.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Tablet-format transfer must preserve the mixed history/realtime data in both models.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

testSinkFormat("tablet", false);
}

// TsFile-format transfer must preserve the mixed history/realtime data in both models.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: TsFile-format transfer must preserve the mixed history/realtime data in both models.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

testSinkFormat("tsfile", false);
}

// Asynchronously loading transferred TsFiles must eventually expose every expected table row

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Asynchronously loading transferred TsFiles must eventually expose every expected table row and tree point.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

testSinkFormat("tsfile", true);
}

// Hybrid-format transfer must preserve all history/realtime rows across the tested flush

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Hybrid-format transfer must preserve all history/realtime rows across the tested flush boundaries.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

TestUtils.executeNonQueryWithRetry(senderEnv, "flush");
TestUtils.executeNonQueryWithRetry(receiverEnv, "flush");
};
try (final RetryFlushHandler handleFailure = new RetryFlushHandler(null);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

// Writing back to database Test1 with forwarding disabled must produce all 400 table rows

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Writing back to database Test1 with forwarding disabled must produce all 400 table rows without a forwarding cycle.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// Transfer 200 rows, restart both clusters, then require all 300 rows after another write.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Transfer 200 rows, restart both clusters, then require all 300 rows after another write.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

TableModelUtils.assertData("test", "test", 0, 500, receiverEnv, handleFailure);
}

// An unprivileged user may list pipes but cannot manage pipes/plugins; after granting SYSTEM,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: An unprivileged user may list pipes but cannot manage pipes/plugins; after granting SYSTEM, lifecycle operations succeed and plugin installation still validates its URI.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

MultiEnvFactory.createEnv(2);
senderEnv = MultiEnvFactory.getEnv(0);
receiverEnv = MultiEnvFactory.getEnv(1);
public class IoTDBPipePermissionIT extends AbstractPipeTableModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

PIPE_META_HEARTBEAT_INTERVAL_SECONDS + 2L;

@Override
protected void setupConfig() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configure a 3-second metadata heartbeat explicitly on both clusters so the unauthorized-transfer observation window covers that heartbeat plus its documented 2-second margin, independently of template defaults.


senderEnv.initClusterEnvironment();
receiverEnv.initClusterEnvironment(3, 3);
@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

CompletableFuture.runAsync(() -> receiverEnv.initClusterEnvironment(3, 3)));
}

// A source user sees only its own pipe and transfers schema/rows only after the required grants.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A source user sees only its own pipe and transfers schema/rows only after the required grants.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// Receiver grants unblock queued rows; skipif excludes a second database until it is authorized.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Receiver grants unblock queued rows; skipif excludes a second database until it is authorized.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

import org.apache.iotdb.it.env.cluster.node.DataNodeWrapper;
import org.apache.iotdb.it.framework.IoTDBTestRunner;
import org.apache.iotdb.itbase.category.MultiClusterIT2DualTableManualBasic;
import org.apache.iotdb.itbase.env.BaseEnv;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

MultiEnvFactory.createEnv(2);
senderEnv = MultiEnvFactory.getEnv(0);
receiverEnv = MultiEnvFactory.getEnv(1);
// These tests cover consensus/topology combinations. Many CPU-scaled tiny

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bound the functional test's DataRegion allocation to its database/topology needs instead of the host CPU count, retaining replica coverage while avoiding excessive small-region WAL/load work in the IT heap.

Use a smaller schema-analysis reservation for these small test TsFiles so concurrent loads fit within the IT node's bounded memory; preserve the data and failover assertions.


senderEnv.initClusterEnvironment(configNodesNum, dataNodesNum);
receiverEnv.initClusterEnvironment(configNodesNum, dataNodesNum);
executeOnBothEnvs(env -> env.initClusterEnvironment(configNodesNum, dataNodesNum));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

}

// Test that receiver will not load data when table exists but TAG columns mismatch
// Historical files whose TAG columns disagree with the receiver must fail schema validation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Historical files whose TAG columns disagree with the receiver must fail schema validation without changing the receiver's existing rows. Observe that failure before checking the rows.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

// incoming TAG columns
@Test
public void testReceiverAutoExtendIdColumn() throws Exception {
final String database = "load_extend_tag";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Test that receiver can load data when table exists and existing TAG columns are the prefix of incoming TAG columns

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

// existing TAG columns
@Test
public void testLoadWhenIncomingIdColumnsArePrefixOfExisting() throws Exception {
final String database = "load_prefix_tag";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Test that receiver can load data when table exists and incoming TAG columns are the prefix of existing TAG columns

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

}
}

// Verify a table deleted before historical capture is not recreated on the receiver.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Verify a table deleted before historical capture is not recreated on the receiver.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

}
}

// A receiver user with CREATE but no INSERT privilege must reject the historical file and leave

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A receiver user with CREATE but no INSERT privilege must reject the historical file and leave its table absent. Wait for that specific denial before checking the catalog.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

}
}

private void awaitReceiverLoadFailure(final String database, final String expectedMessage) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poll immediately and then every 100 ms for the observable condition while keeping a bounded wait and preserving the scenario assertion.

@RunWith(IoTDBTestRunner.class)
@Category({MultiClusterIT2DualTableManualBasic.class})
public class IoTDBTablePatternFormatIT extends AbstractPipeTableModelDualManualIT {
public class IoTDBTablePatternFormatIT extends AbstractPipeTableModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

public class IoTDBTablePatternFormatIT extends AbstractPipeTableModelDualManualIT {
public class IoTDBTablePatternFormatIT extends AbstractPipeTableModelSharedClusterIT {

@Rule public final TestName testName = new TestName();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

super.setUp();
}

// A table-name pattern must replicate both matching tables' history and realtime rows [0, 200).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A table-name pattern must replicate both matching tables' history and realtime rows [0, 200).

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

// A table-name pattern with time bounds [0, 49] must load only matching historical rows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A table-name pattern with time bounds [0, 49] must load only matching historical rows.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

Collections.singletonList("create table test()"), "alter table test add column a tag");
}

// Adding an existing tag to the receiver's view must be treated as an idempotent conflict.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Adding an existing tag to the receiver's view must be treated as an idempotent conflict.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter view test add column a tag");
}

// Reapplying TTL=100 to a table must preserve pipe progress after the duplicate ALTER.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Reapplying TTL=100 to a table must preserve pipe progress after the duplicate ALTER.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter table test set properties ttl=100");
}

// Reapplying TTL=100 to a view must preserve pipe progress after the duplicate ALTER.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Reapplying TTL=100 to a view must preserve pipe progress after the duplicate ALTER.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter view test set properties ttl=100");
}

// Dropping an attribute already removed on the receiver must not block subsequent events.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping an attribute already removed on the receiver must not block subsequent events.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter table test drop column b");
}

// Dropping a view field already removed on the receiver must allow sync to continue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping a view field already removed on the receiver must allow sync to continue.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter view test drop column s1");
}

// Dropping a table already absent on the receiver must not stall later database creation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping a table already absent on the receiver must not stall later database creation.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

testTableConfigIdempotent(Collections.singletonList("create table test()"), "drop table test");
}

// Dropping a view already absent on the receiver must not stall later database creation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping a view already absent on the receiver must not stall later database creation.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"drop view test");
}

// Replaying a view-column rename from a to b must allow subsequent events to be applied.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Replaying a view-column rename from a to b must allow subsequent events to be applied.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter view test rename column a to b");
}

// Replaying a view rename from test to test1 must allow subsequent events to be applied.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Replaying a view rename from test to test1 must allow subsequent events to be applied.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

"alter view test rename to test1");
}

// Creating the same user independently on both clusters must not block pipe progress.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Creating the same user independently on both clusters must not block pipe progress.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

receiverEnv.initClusterEnvironment();
}

// The async thrift sink with stream extraction, batching, and Snappy must preserve tree and

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The async thrift sink with stream extraction, batching, and Snappy must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

doTest("iotdb-thrift-connector", "stream", true, "snappy");
}

// The async thrift sink with batch extraction and Snappy/LZMA2 compression must preserve tree

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The async thrift sink with batch extraction and Snappy/LZMA2 compression must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

doTest("iotdb-thrift-connector", "batch", true, "snappy, lzma2");
}

// The sync thrift sink with stream extraction and repeated Snappy compression must preserve

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The sync thrift sink with stream extraction and repeated Snappy compression must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

doTest("iotdb-thrift-sync-connector", "stream", false, "snappy, snappy");
}

// The sync thrift sink with batch extraction and gzip/Zstd compression must preserve tree and

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The sync thrift sink with batch extraction and gzip/Zstd compression must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

doTest("iotdb-thrift-sync-connector", "batch", true, "gzip, zstd");
}

// The air-gap sink with stream extraction and LZMA2/LZ4 compression must preserve tree and

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The air-gap sink with stream extraction and LZMA2/LZ4 compression must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

doTest("iotdb-air-gap-connector", "stream", false, "lzma2, lz4");
}

// The air-gap sink with batch extraction and LZMA2 compression must preserve tree and table

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The air-gap sink with batch extraction and LZMA2 compression must preserve tree and table data.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

TestUtils.executeNonQueryWithRetry(senderEnv, "flush");
TestUtils.executeNonQueryWithRetry(receiverEnv, "flush");
};
final String database = database("test");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

// Accept Zstd levels -131072, 3, and 22, reject levels -131073 and 23 on the same connection,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Accept Zstd levels -131072, 3, and 22, reject levels -131073 and 23 on the same connection, retain only the three valid pipes, and replicate data through a valid pipe.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Keep flush-retry clients in one handler owned by the scenario, so repeated assertion polls reuse Sessions and close them before cluster reset.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

// successful first assertion poll needs no retry clients.
private final ISession[] sessions = new ISession[2];

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close scenario-owned clients in the lifecycle that owns them, retaining cleanup even on failure; this avoids carrying connections into a reset/restarted cluster.

@Override
protected void setupConfig() {
super.setupConfig();
senderEnv

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply the scenario-specific settings before node startup, retaining its consensus/replica requirements while inheriting the shared fixture's bounded resource defaults where applicable.

senderEnv.getConfig().getCommonConfig().setPipeAutoSplitFullEnabled(true);
}

// Realtime-only replication starts after timestamp 0 is written and must transfer timestamp 1.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Realtime-only replication starts after timestamp 0 is written and must transfer timestamp 1.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// Tablet sink format transfers realtime points across initial and recreated pipe instances.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Tablet sink format transfers realtime points across initial and recreated pipe instances.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

testSinkFormat("tablet");
}

// TsFile sink format transfers realtime points across initial and recreated pipe instances.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: TsFile sink format transfers realtime points across initial and recreated pipe instances.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

TestUtils.executeNonQueries(
senderEnv,
Arrays.asList("insert into root.vehicle.d0(time, s1) values (1, 1)", "flush"),
isolateDatabasePaths(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// The legacy sink must transfer the first row in its own database even when earlier tests left

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: The legacy sink must transfer the first row in its own database even when earlier tests left asynchronous regions pending cleanup.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}
}

// A history and realtime log pipe must create receiver series from source DDL and inserts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A history and realtime log pipe must create receiver series from source DDL and inserts.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

});
}

// A history and realtime file pipe must create receiver series from source DDL and inserts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A history and realtime file pipe must create receiver series from source DDL and inserts.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

});
}

// A source path pattern limits receiver auto-creation to the two matching devices.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A source path pattern limits receiver auto-creation to the two matching devices.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

// For pattern parse
"insert into root.ln.wf01.wt11(time, redundant_data) values (20000, -1)",
"flush"),
isolateDatabasePaths(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

receiverEnv = MultiEnvFactory.getEnv(1);
public class IoTDBPipeSourceIT extends AbstractPipeTreeModelSharedClusterIT {

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply the scenario-specific settings before node startup, retaining its consensus/replica requirements while inheriting the shared fixture's bounded resource defaults where applicable.

receiverEnv.initClusterEnvironment();
}

// Each valid history/realtime time-boundary parameter creates a pipe successfully; reuse one

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Each valid history/realtime time-boundary parameter creates a pipe successfully; reuse one tree-model JDBC connection for all ten CREATE PIPE statements and retain count checkpoints.

Reuse one method-owned SQL statement across the source-parameter matrix. Keep each legal configuration and invalid-bound/history-realtime rejection, and check pipe counts so rejected statements cannot leave a pipe behind on the shared cluster.

assertPipeCount(10);
}

// Invalid time bounds and mutually disabled history/realtime options must be rejected without

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Invalid time bounds and mutually disabled history/realtime options must be rejected without leaving any pipe behind; expected failures share one connection across all parameter checks.

Reuse one method-owned SQL statement across the source-parameter matrix. Keep each legal configuration and invalid-bound/history-realtime rejection, and check pipe counts so rejected statements cannot leave a pipe behind on the shared cluster.

}
}

// Replicate only timestamps in [2000, 4000] from history and realtime events, including a Tablet

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Replicate only timestamps in [2000, 4000] from history and realtime events, including a Tablet with unused tail slots. After data completion, out-of-range devices must have no receiver series.

Wait for the committed full-flush barrier before checking the exact receiver data; this avoids paying a fixed observation window for a completed data-only transfer.

}
}

// A start-time bound must not re-enable history: historical series stay absent after initial

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A start-time bound must not re-enable history: historical series stay absent after initial completion and after realtime events, while timestamps at and above 2000 are transferred.

Wait for the committed full-flush barrier before checking the exact receiver data; this avoids paying a fixed observation window for a completed data-only transfer.

@RunWith(IoTDBTestRunner.class)
@Category({MultiClusterIT2DualTreeAutoBasic.class})
public class IoTDBPipeSwitchStatusIT extends AbstractPipeDualTreeModelAutoIT {
public class IoTDBPipeSwitchStatusIT extends AbstractPipeTreeModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

public void setUp() {
super.setUp();
}
public class IoTDBPipeSyntaxIT extends AbstractPipeTreeModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

}
public class IoTDBPipeSyntaxIT extends AbstractPipeTreeModelSharedClusterIT {

// Legal identifiers, including quoted numbers and reserved words, must create and drop cleanly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Legal identifiers, including quoted numbers and reserved words, must create and drop cleanly.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

// Reordering source and sink parameters must retain validation of the disabled history mode.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Reordering source and sink parameters must retain validation of the disabled history mode.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

}
}

// Putting the sink before the source must fail and leave no pipe registered.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Putting the sink before the source must fail and leave no pipe registered.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

private ISession receiverSession;
private boolean clustersReady;

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

clustersReady = true;
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply the scenario-specific settings before node startup, retaining its consensus/replica requirements while inheriting the shared fixture's bounded resource defaults where applicable.

receiverEnv.initClusterEnvironment();
}

// Both IOT-consensus clusters must merge 600 disjoint rows despite automatic schema conflicts,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Both IOT-consensus clusters must merge 600 disjoint rows despite automatic schema conflicts, including the final 200 rows written after both clusters restart.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

expectedResSet);
}

// Bidirectional pipes must auto-extend a shared template for s1 and merge 400 rows on both sides.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Bidirectional pipes must auto-extend a shared template for s1 and merge 400 rows on both sides.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

expectedResSet);
}

// A manually created receiver series must acquire the sender tag and attribute when auto-created

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A manually created receiver series must acquire the sender tag and attribute when auto-created schema events race with the existing series.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

}
}

// Historical sync must preserve an aligned template binding and all three activated devices,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Historical sync must preserve an aligned template binding and all three activated devices, including a device activated by a negative timestamp and a later template extension.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

}
}

private String isolateDatabasePaths(final String sql) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

"$0_" + Integer.toUnsignedString(testName.getMethodName().hashCode(), 16));
}

private ISession writeSession(final BaseEnv env) throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

return receiverSession;
}

private void insertData(final BaseEnv env, final int start, final int end, final boolean flush)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

private void closeWriteSessions() throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

Collections.singleton("2,"));
}

// Reapplying schema/data region limits must allow a second database to sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Reapplying schema/data region limits must allow a second database to sync.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Collections.singleton("2,"));
}

// Dropping an already absent database must allow the next creation to sync (one remaining).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping an already absent database must allow the next creation to sync (one remaining).

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Collections.singleton("1,"));
}

// Creating an existing user must not block the subsequent database event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Creating an existing user must not block the subsequent database event.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

Collections.singleton("1,"));
}

// Creating an existing role must not block the subsequent database event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Creating an existing role must not block the subsequent database event.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Collections.singleton("1,"));
}

// Granting an already granted role membership must allow later database events to sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Granting an already granted role membership must allow later database events to sync.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

Collections.singleton("1,"));
}

// Revoking already absent user path privileges must allow the next database event to sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Revoking already absent user path privileges must allow the next database event to sync.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

Collections.singleton("1,"));
}

// Revoking already absent role privileges must allow the next database event to sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Revoking already absent role privileges must allow the next database event to sync.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Collections.singleton("1,"));
}

// Revoking an already absent role membership must allow later database events to sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Revoking an already absent role membership must allow later database events to sync.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

Collections.singleton("1,"));
}

// Dropping an already absent user must not block the subsequent database event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping an already absent user must not block the subsequent database event.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

Collections.singleton("1,"));
}

// Dropping an already absent role must not block the subsequent database event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Dropping an already absent role must not block the subsequent database event.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

return value.replace("root.db", databasePath());
}

private List<String> isolateDatabasePaths(final List<String> values) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.


@RunWith(IoTDBTestRunner.class)
@Category({MultiClusterIT2DualTreeAutoEnhanced.class})
public class IoTDBPipeTypeConversionSemanticIT extends AbstractPipeDualTreeModelAutoIT {
public class IoTDBPipeTypeConversionSemanticIT extends AbstractPipeTreeModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

@@ -81,6 +69,13 @@ protected void setupConfig() {
.setSchemaReplicationFactor(1);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply the scenario-specific settings before node startup, retaining its consensus/replica requirements while inheriting the shared fixture's bounded resource defaults where applicable.

@@ -81,6 +69,13 @@ protected void setupConfig() {
.setSchemaReplicationFactor(1);
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

@@ -41,6 +41,9 @@
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

@@ -81,8 +84,7 @@ public void setUp() {

senderEnv.getConfig().getConfigNodeConfig().setLeaderDistributionPolicy("HASH");

senderEnv.initClusterEnvironment();
receiverEnv.initClusterEnvironment();
executeOnBothEnvs(env -> env.initClusterEnvironment());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

// the following deletions performed on the new leader can not delete anything
// wait for a while to increase the chance that data has been synced to the new leader
Thread.sleep(5000);
// Adding devices may create a new region and trigger leader re-balancing. Wait for the

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poll immediately and then every 100 ms for the observable condition while keeping a bounded wait and preserving the scenario assertion.

public void setUp() {
super.setUp();
}
public class PipeNowFunctionIT extends AbstractPipeTreeModelSharedClusterIT {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.


// CREATE/ALTER PIPE through RPC accepts "now" for prefixed and unprefixed history/source time

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: CREATE/ALTER PIPE through RPC accepts "now" for prefixed and unprefixed history/source time bounds; each created pipe remains visible and RUNNING until explicitly stopped.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

@@ -183,6 +178,8 @@ public void testPipeNowFunction() throws Exception {
}
}

// Tree-model CREATE/ALTER PIPE must accept now in prefixed and unprefixed history/source time

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Tree-model CREATE/ALTER PIPE must accept now in prefixed and unprefixed history/source time attributes; execute all six variants in order on the same connection.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.


senderEnv.initClusterEnvironment();
receiverEnv.initClusterEnvironment(3, 3);
@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overlap only independent environment/node actions and join every task, propagating failures before subsequent verification or teardown.

}

// Both thrift sinks must authenticate as the receiver user and replicate schema, rows and users.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Both thrift sinks must authenticate as the receiver user and replicate schema, rows and users.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

@@ -125,7 +126,7 @@ private void testWithSink(final String sink, final String suffix) throws Excepti
TestUtils.executeNonQueries(
senderEnv,
Arrays.asList(
"create user " + sourceUser + " 'passwd123456'",
"create user " + sourceUser + " 'IoTDB!Test2024'",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

@@ -202,17 +203,34 @@ private void assertUserEventuallyExists(final String username) {
});
}

// A receiver user with read-only privileges must receive neither source users nor timeseries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A receiver user with read-only privileges must receive neither source users nor timeseries.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

}
}

// Granting source READ must unblock aligned rows; the two-stage processor uses that source user.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Granting source READ must unblock aligned rows; the two-stage processor uses that source user.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -344,8 +363,8 @@ private void testCountPointProcessorUsesSourceCredentials() throws Exception {
sinkAttributes.put("sink.batch.enable", "false");
sinkAttributes.put("sink.ip", receiverDataNode.getIp());
sinkAttributes.put("sink.port", Integer.toString(receiverDataNode.getPort()));
sinkAttributes.put("sink.user", "root");
sinkAttributes.put("sink.password", "root");
sinkAttributes.put("sink.user", SessionConfig.DEFAULT_USER);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

@@ -373,6 +392,7 @@ private void testCountPointProcessorUsesSourceCredentials() throws Exception {
Collections.singleton("3,"));
}

// Missing/wrong source credentials fail; schema and rows stay filtered until grants permit them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Missing/wrong source credentials fail; schema and rows stay filtered until grants permit them.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -604,6 +624,8 @@ public void testSourcePermission() {
}
}

// Wrong passwords fail creation. After each sender restart, correcting the source password

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Wrong passwords fail creation. After each sender restart, correcting the source password or restoring the user's password must resume replication of all three rows.

Use a policy-compliant password for the normal test-created account and its matching client references, keeping the intentional wrong-password paths unchanged.

@@ -57,6 +57,7 @@ protected void setupConfig() {
receiverEnv.getConfig().getCommonConfig().setPipeConnectorRequestSliceThresholdBytes(4);
}

// A ten-row tablet exceeding the four-byte request threshold must arrive intact after reassembly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A ten-row tablet exceeding the four-byte request threshold must arrive intact after reassembly.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -313,7 +314,7 @@ private void prepareReqAutoSliceTest(
createPipe(senderSession, true);
} else {
createPipe(senderSession, false);
Thread.sleep(2000);
// Pipe creation has completed; verify() waits for delivery of the sliced requests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

@@ -19,25 +19,37 @@

package org.apache.iotdb.pipe.it.single;

import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

@ClassRule
public static final ExternalResource CLUSTER =
new ExternalResource() {
@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release class-owned resources at the owning lifecycle boundary, including partially initialized environments and failure paths.

.setPipeAutoSplitFullEnabled(false);
// 1C1D to directly show the remaining count
env.initClusterEnvironment(1, 1);
public void setUp() throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

env = sharedEnv;
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reset or release the scenario-owned state at its lifecycle boundary, stopping replication before deleting shared state so later methods remain independent.

}
}

private static void dropPipes(final BaseEnv env) throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

}
}

private static void resetCatalog(final BaseEnv env) throws SQLException {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

@@ -23,10 +23,14 @@
import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
import org.apache.iotdb.commons.cluster.NodeStatus;
import org.apache.iotdb.commons.conf.CommonDescriptor;
import org.apache.iotdb.commons.pipe.agent.task.meta.PipeStaticMeta;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep compatible methods on class-owned clusters and isolate/reset their test state, so a method pays for the lifecycle required by its scenario rather than rebuilding the same topology.

@ClassRule
public static final ExternalResource CLUSTER =
new ExternalResource() {
@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release class-owned resources at the owning lifecycle boundary, including partially initialized environments and failure paths.

}
};

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

env = sharedEnv;
}

@Override

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the configured default administrator identity for the ordinary source/sink setup, keeping edition-specific defaults consistent without changing intentional permission-rejection accounts.

Keep the ConfigNode client in the scenario resource scope alongside its JDBC statement, so repeated metadata checks reuse the same client and cleanup cannot leak it into the next shared-cluster case.

Poll immediately and then every 100 ms for the observable condition while keeping a bounded wait and preserving the scenario assertion.

@@ -417,16 +451,15 @@ private void testWithTimeColumn(
try (final Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
final Statement statement = connection.createStatement()) {
statement.execute(String.format("create database if not exists %s", SchemaConfig.DATABASE_0));
statement.execute(String.format("use %s", SchemaConfig.DATABASE_0));
statement.execute(String.format("create database if not exists %s", database));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the caller's scenario database consistently for LOAD, schema comparison and exact row-count checks, then drop that database. Each shared-cluster case keeps its original schema/time-column assertions.

@@ -499,6 +532,10 @@ private String convert2TableSQL(
return tableCreation;
}

/**
* Load 20 rows at timestamps 1,000 through 20,000 with both fields null; expect all timestamps to

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -599,11 +635,10 @@ public void testLoadSparseBatchedAlignedTsFile() throws Exception {
try (final Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
final Statement statement = connection.createStatement()) {
statement.execute(String.format("create database if not exists %s", SchemaConfig.DATABASE_0));
statement.execute(String.format("use %s", SchemaConfig.DATABASE_0));
statement.execute(String.format("create database if not exists %s", database));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -31,7 +31,8 @@

import org.apache.tsfile.file.metadata.IDeviceID;
import org.junit.After;
import org.junit.Before;
import org.junit.AfterClass;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

@@ -61,8 +62,9 @@ public class IoTDBRestartTableIT {

private final Logger logger = LoggerFactory.getLogger(IoTDBRestartTableIT.class);

@Before
public void setUp() throws Exception {
// Reuse the fixed WAL/schema configuration while retaining every scenario's recovery restarts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bound the functional test's DataRegion allocation to its database/topology needs instead of the host CPU count, retaining replica coverage while avoiding excessive small-region WAL/load work in the IT heap.

@@ -82,8 +84,8 @@ public void setUp() throws Exception {
EnvFactory.getEnv().initClusterEnvironment();
}

@After
public void tearDown() throws Exception {
@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown. Restore the fixture's WAL/schema-consensus overrides after shutdown.

@@ -92,6 +94,18 @@ public void tearDown() throws Exception {
.setSchemaRegionConsensusProtocolClass("org.apache.iotdb.consensus.simple.SimpleConsensus");
}

@After

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop the table recovery database after each method, preserving each scenario's internal recovery restarts while avoiding a new cluster for identical initial configuration.

}

/**
* Write flushed and unflushed table rows across two restarts, then verify all three timestamps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -139,6 +153,10 @@ public void testRestart() throws SQLException {
}
}

/**
* Recover three points and verify the remaining values after two range deletions, with a flush

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -255,6 +273,7 @@ private static void assertRestartTableData(Statement statement, String... target
}
}

/** Write rows across two restarts and verify the time filter returns only the row at time 4. */

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -27,8 +27,9 @@

import org.apache.tsfile.enums.TSDataType;
import org.junit.After;
import org.junit.AfterClass;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

@@ -57,13 +59,32 @@
public class IoTDBSimpleQueryTableViewIT {
private static final String DATABASE_NAME = "test";

@Before
public void setUp() throws Exception {
@BeforeClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize the class's fixed node configuration once for compatible methods. Per-method data/state preparation or isolated namespaces preserve the individual scenarios without repeated cluster startup.

EnvFactory.getEnv().initClusterEnvironment();
}

@After
public void tearDown() throws Exception {
public void tearDown() throws SQLException {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this method's test-owned data/schema after execution while retaining class-owned nodes. The next case prepares its own input and cannot inherit the previous result.

}
}

@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

@@ -19,16 +19,18 @@

package org.apache.iotdb.relational.it.query.view.recent;

import org.apache.iotdb.isession.ISession;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.


public IoTDBTableViewQueryIT(boolean aligned, boolean flush) {
this.aligned = aligned;
this.flush = flush;
// Isolate both models so parameter cases can share the cluster without reusing fixture state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

}

@Before
public void setUp() throws Exception {
@BeforeClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prepare the scenario's configuration/state before execution. Compatible classes retain class-owned nodes; protocol/topology matrix cases retain their required separate lifecycle and bounded region budget.

prepareData(new String[] {"flush"});
}

@Before

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}

@After
public void tearDown() throws Exception {
@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// Compare tree-backed views with a physical table for aligned/non-aligned, flushed/memory

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Compare tree-backed views with a physical table for aligned/non-aligned, flushed/memory fixtures; nulls, TTL, filters, fills, joins and aggregations must keep the same results.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

private String device1;
private String device2;

@BeforeClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize the class's fixed node configuration once for compatible methods. Per-method data/state preparation or isolated namespaces preserve the individual scenarios without repeated cluster startup.

EnvFactory.getEnv().getConfig().getCommonConfig().setMaxDegreeOfIndexNode(3);
EnvFactory.getEnv().initClusterEnvironment();
}

@After
public void tearDown() throws Exception {
@Before

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

device2 = database + ".d2";
}

@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// Insert and flush 100 aligned tablet rows, then verify both typed columns and timestamps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Insert and flush 100 aligned tablet rows, then verify both typed columns and timestamps.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -113,174 +130,181 @@ private void insertAlignedRecord(ISession session, String deviceId)
}
}

// Insert 100 typed aligned records and verify s2 after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Insert 100 typed aligned records and verify s2 after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}

// Insert 100 aligned string records and verify their numeric s2 values after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Insert 100 aligned string records and verify their numeric s2 values after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}

// Batch aligned string records and verify all 100 converted s2 values after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Batch aligned string records and verify all 100 converted s2 values after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}

// Batch typed aligned records and verify all 100 s2 values after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Batch typed aligned records and verify all 100 s2 values after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}

// Batch aligned records for one device and verify all 100 s2 values after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Batch aligned records for one device and verify all 100 s2 values after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}

// Mix aligned and non-aligned devices, then verify the aligned s2 projection after flush.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Mix aligned and non-aligned devices, then verify the aligned s2 projection after flush.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

EnvFactory.getEnv().initClusterEnvironment();
}

@Before
public void setUp() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the scenario-specific database consistently in writes, pipe filters and expectations so asynchronous cleanup from a previous method cannot contaminate the shared cluster.

@After
public void tearDown() throws Exception {
dropDatabaseIfExists(database);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this method's test-owned data/schema after execution while retaining class-owned nodes. The next case prepares its own input and cannot inherit the previous result.

dropDatabaseIfExists(database);
}

@AfterClass

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Destroy the class-owned cluster after all compatible scenarios finish, so individual methods avoid repeated process startup/teardown.

EnvFactory.getEnv().cleanClusterEnvironment();
}

private void dropDatabaseIfExists(String databasePath) throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

}
}

// A tablet with incompatible value types fails without returning partial rows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: A tablet with incompatible value types fails without returning partial rows.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -126,13 +157,14 @@ public void testInsertByStrAndSelectFailedData() {
}
}

// DATE, TIMESTAMP, BLOB, and STRING values round-trip through typed record insertion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: DATE, TIMESTAMP, BLOB, and STRING values round-trip through typed record insertion.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -194,13 +227,14 @@ public void testInsertRecord() {
}
}

// Text-form records are parsed into DATE, TIMESTAMP, BLOB, and STRING values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Text-form records are parsed into DATE, TIMESTAMP, BLOB, and STRING values.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -263,11 +298,12 @@ public void testInsertStrRecord() {
}
}

// Tablet insertion preserves typed values across both populated batches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Tablet insertion preserves typed values across both populated batches.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -342,13 +379,14 @@ public void testInsertTablet() {
}
}

// Session operations before open and after close consistently report a closed session.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Session operations before open and after close consistently report a closed session.

Reuse the scenario-owned Session for repeated writes/flushes, with closure before cluster reset or restart so recovery uses fresh clients.

@@ -37,8 +37,9 @@
import org.apache.iotdb.session.pool.SessionPool;

import org.awaitility.Awaitility;
import org.junit.After;
import org.junit.AfterClass;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain the existing scenario lifecycle while adding the supporting imports/state for the bounded polling, Session reuse or diagnostics changes in this class.

@@ -375,7 +417,7 @@ private void write10Data(ISessionPool pool, boolean failWhenThrowException) {
for (int i = 0; i < 10; i++) {
try {
pool.insertRecord(
"root.sg1.d1",
database + ".d1",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write the same ten single-record requests into the method's isolated database, preserving both normal writes and intentional failure handling in the pool scenarios.

@@ -389,13 +431,14 @@ private void write10Data(ISessionPool pool, boolean failWhenThrowException) {
}
}

// Closing a pool rejects later writes with the closed-pool error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Closing a pool rejects later writes with the closed-pool error.

Keep this case's expected result and regression boundary explicit while adapting its shared fixture/client calls; the exact assertions remain at this method.

@@ -410,6 +453,7 @@ public void testClose() {
// session.
}

// Pool setters update their corresponding observable configuration values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Pool setters update their corresponding observable configuration values.

Record the preserved input and expected result explicitly for this case so its regression boundary remains reviewable after the fixture/runtime changes.

@@ -47,6 +47,8 @@ public class IoTDBRegionReadyAfterRestartTableIT {

@BeforeClass
public static void setUpClass() {
// Keep the pre-restart row durable independently of Windows process-stop timing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use SYNC WAL for the upstream Region-readiness restart regression. On Windows the process stop can terminate before ASYNC WAL persists the first acknowledged row; a faster stop/start must not make input durability depend on scheduling. Keep the original 1C3D topology and all Running-region and exact two-row assertions without adding a flush to the recovery path.

@@ -55,6 +57,7 @@ public static void tearDownClass() {
EnvFactory.getEnv().cleanClusterEnvironment();
}

// Persist one row, restart all nodes, then require Running Regions and both rows after a write.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scenario: Persist one row, restart all nodes, then require Running Regions and both rows after a write.

Document the preserved upstream regression: persist a first row, restart all nodes, verify every reported Region is Running, then write and read both rows exactly. Only the fixture makes durability explicit; the test body and its readiness/recovery checks are unchanged.


DataNodeConfig setMqttPayloadFormatter(String mqttPayloadFormatter);

DataNodeConfig setMqttHandlerPoolSize(int mqttHandlerPoolSize);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expose the MQTT callback pool size through the IT configuration interface so payload fixtures can request serialized CONNECT/PUBLISH handling without downcasting the environment. The setting is opt-in; the product default is unchanged.

}

@Override
public DataNodeConfig setMqttHandlerPoolSize(int mqttHandlerPoolSize) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Persist mqtt_handler_pool_size in each launched DataNode configuration. One callback worker makes CONNECT registration complete before the queued single-publish fixture input runs. The four full-class/profile runs pass 18 cases, and all eight DataNode logs show mqttHandlerPoolSize=1.

}

@Override
public DataNodeConfig setMqttHandlerPoolSize(int mqttHandlerPoolSize) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep the new IT setting a no-op for externally managed nodes, matching the existing MQTT configuration methods: this adapter does not rewrite or restart a remote deployment. Both configuration implementations compile through the four targeted reactor verifications.

baseEnv.getConfig().getDataNodeConfig().setMqttPayloadFormatter(FORMATTER);
// Serialize CONNECT and PUBLISH callbacks so the session is registered before each
// payload's single publish is processed.
baseEnv.getConfig().getDataNodeConfig().setMqttHandlerPoolSize(1);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serialize broker CONNECT and PUBLISH callbacks for the JSON payload scenarios. With two callback workers a first PUBLISH can run before the session map is populated and be discarded, which is consistent with the Windows test timeout. A controlled probe of the actual Moquette executor reproduces callback overtaking with two workers and ordered execution with one; the complete seven-case class passes on both one and three DataNodes. The existing single publish, illegal-measurement input and data assertions are retained.

BaseEnv baseEnv = EnvFactory.getEnv();
baseEnv.getConfig().getDataNodeConfig().setEnableMQTTService(true);
baseEnv.getConfig().getDataNodeConfig().setMqttPayloadFormatter(FORMATTER);
// Serialize CONNECT and PUBLISH callbacks so the session is registered before each
// payload's single publish is processed.
baseEnv.getConfig().getDataNodeConfig().setMqttHandlerPoolSize(1);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply the same CONNECT-before-PUBLISH setup to both table payload scenarios, which also publish each input once. One broker callback worker establishes their required session lifecycle order while preserving the existing isolated databases and payload assertions. Both complete cases pass under TableSimpleIT and TableClusterIT.

.dropPipeExtended(new TDropPipeReq(pipe.getId()).setIsTableModel(false))
.getCode();
// An in-flight DROP PIPE can finish after the SHOW PIPE snapshot.
if (statusCode != TSStatusCode.PIPE_NOT_EXIST_ERROR.getStatusCode()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow only PIPE_NOT_EXIST_ERROR when a pipe in the SHOW snapshot is already absent at cleanup DROP. The Ubuntu CI failure interrupted cleanup before database deletion, causing the next case to fail on the retained root.sg. Keeping the success assertion for every other status preserves unexpected failures. All five basic-shard classes pass locally (32 cases); a controlled RPC probe runs this actual helper before and after, checking six cases including unrelated status/transport failures and system-pipe exclusion. The probe does not start an IoTDB server.

TSStatusCode.SUCCESS_STATUS.getStatusCode(),
client
.dropPipeExtended(new TDropPipeReq(pipe.getId()).setIsTableModel(tableModel))
.getCode());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Tolerate pipes removed after the snapshot. showPipe is a snapshot; while this loop processes it, the automatic-drop procedure can remove a listed pipe. A subsequent dropPipeExtended then returns PIPE_NOT_EXIST_ERROR for that entry, turning successful cleanup into a flaky test failure. Please ignore only that status here and in IoTDBPipeAutoDropIT.dropTablePipes, as the tree shared-cluster cleanup already does.

EnvFactory.getEnv().cleanClusterEnvironment();
}

// A wildcard CQ copies only readable fields, observes revoke/grant, and rejects a deleted owner.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve the upstream CQ permission regression while merging its required RPC/status imports with the existing per-DataNode result polling. Document this wrapper's actual permission scenario; retain the original resampling comment at testCQExecution1. The upstream RPC sequence and assertions are unchanged. The complete CQ execution classes and CQ validation class pass in both SimpleIT and ClusterIT.

testCQExecutionWithReadDataPermission(false, false);
}

// An aligned wildcard CQ filters unreadable fields and rechecks owner permissions per window.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve the upstream CQ permission regression while merging its required RPC/status imports with the existing per-DataNode result polling. Document this wrapper's actual permission scenario; retain the original resampling comment at testCQExecution1. The upstream RPC sequence and assertions are unchanged. The complete CQ execution classes and CQ validation class pass in both SimpleIT and ClusterIT.

testCQExecutionWithReadDataPermission(true, false);
}

// Explicit CQ targets reject missing source access and succeed when that access is restored.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve the upstream CQ permission regression while merging its required RPC/status imports with the existing per-DataNode result polling. Document this wrapper's actual permission scenario; retain the original resampling comment at testCQExecution1. The upstream RPC sequence and assertions are unchanged. The complete CQ execution classes and CQ validation class pass in both SimpleIT and ClusterIT.

testCQExecutionWithReadDataPermission(false, true);
}

// Aligned explicit CQ targets retain positional mapping across permission revoke/grant retries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve the upstream CQ permission regression while merging its required RPC/status imports with the existing per-DataNode result polling. Document this wrapper's actual permission scenario; retain the original resampling comment at testCQExecution1. The upstream RPC sequence and assertions are unchanged. The complete CQ execution classes and CQ validation class pass in both SimpleIT and ClusterIT.

client
.dropPipeExtended(new TDropPipeReq(pipe.getId()).setIsTableModel(tableModel))
.getCode();
// Automatic drop can remove a pipe after the SHOW PIPE snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Treat a user pipe removed by automatic drop after SHOW PIPE as successful cleanup. Ignore only PIPE_NOT_EXIST_ERROR, retain success assertions for every other status, and continue cleaning later pipes and dialects. The real helpers pass 16 controlled RPC checks per version, including unrelated status/transport failures and system-pipe exclusion; both complete Table/Tree AutoDrop classes pass (6 tests).

client
.dropPipeExtended(new TDropPipeReq(pipe.getId()).setIsTableModel(true))
.getCode();
// Automatic drop can remove a pipe after the SHOW PIPE snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Treat a user pipe removed by automatic drop after SHOW PIPE as successful cleanup. Ignore only PIPE_NOT_EXIST_ERROR, retain success assertions for every other status, and continue cleaning later pipes and dialects. The real helpers pass 16 controlled RPC checks per version, including unrelated status/transport failures and system-pipe exclusion; both complete Table/Tree AutoDrop classes pass (6 tests).

@jt2594838
jt2594838 merged commit 5413fd1 into apache:master Nov 13, 2026
38 of 39 checks passed
@jt2594838
jt2594838 deleted the codex/optimize-it-runtime branch November 13, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants