Skip to content

HIVE-26089: ported TestAsyncPbRpcProxy to junit5, front-loaded JVM warmup and un-ignored a test - #6733

Open
konstantinb wants to merge 3 commits into
apache:masterfrom
konstantinb:HIVE-26089
Open

HIVE-26089: ported TestAsyncPbRpcProxy to junit5, front-loaded JVM warmup and un-ignored a test#6733
konstantinb wants to merge 3 commits into
apache:masterfrom
konstantinb:HIVE-26089

Conversation

@konstantinb

@konstantinb konstantinb commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Test-only changes to TestAsyncPbRpcProxy (no production code, no pom changes — JUnit 5 was
already a llap-client test dependency):

  • Ported the class to JUnit 5.
  • Front-loaded the one-time JVM initialization into a @BeforeAll warmUp() with its own
    60-second @Timeout: Mockito mock generation for the two mocked types, logging setup, and
    classloading of the classes under test.
  • Kept the original 5-second per-test budgets as @Timeout(value = 5, threadMode = SEPARATE_THREAD)SEPARATE_THREAD is required for JUnit 4-equivalent preemption
    (Jupiter's default mode only interrupts, which would not stop a hard hang).
  • Un-ignored testSingleInvocationPerNode, disabled since 2022.

This supersedes the approach of #6440 (@Ignore on testMultipleNodes) while restoring the
coverage removed by the 2022 direct-to-master disable
(cc6c02da9).

Why are the changes needed?

Both recorded failures of this class timed out during one-time JVM bootstrap, never in the
code under test: the original HIVE-26089 report died in classloading (ZipFile.getEntry),
and a recent precommit occurrence
(PR 6729, first run)
died inside the first Mockito.mock() call while ByteBuddy generated the mock class. With
surefire's reuseForks=false, every test class starts a cold JVM, so this cost is paid per
class and lands on whichever timed test runs first. The 5s timeouts exist to catch deadlocks
in RequestManager's lock/condition logic (the tests complete in ~0.5s) — they were never
meant to time JVM startup.

The flake reproduces deterministically with no code changes by simulating a starved executor
with a CPU quota — a fresh JVM per run matches reuseForks=false:

# from the repo root, on master (pre-patch)
mvn -pl llap-client test-compile dependency:build-classpath -Dmdep.outputFile=cp.txt -Dmdep.includeScope=test
podman run --rm --cpus=0.15 -v "$PWD:$PWD" -v "$HOME/.m2:$HOME/.m2" docker.io/eclipse-temurin:21-jdk \
  java -cp "$PWD/llap-client/target/test-classes:$PWD/llap-client/target/classes:$(cat cp.txt)" \
  org.junit.runner.JUnitCore org.apache.hadoop.hive.llap.TestAsyncPbRpcProxy

At --cpus=0.15 the unmodified test fails every run (observed 4/4) with the exact CI stack —
TestTimedOutException frozen inside Mockito/ByteBuddy bootstrap at the first mock() call —
and passes from --cpus=0.35 up (docker works the same as podman here).

Splitting the budgets keeps both signals sharp: a 5s per-test timeout still catches a
deadlock, while an init overrun now fails as warmUp() timed out — an environment problem,
clearly distinguished from a test failure.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

  • CPU-quota measurements (fresh JVM per run): unmodified test FAILS 4/4 at 0.15 CPU; with this
    patch both tests PASS at 0.15 and 0.10; at 0.05 (1/20 of a core) the init budget fails the
    class with warmUp() timed out after 60 seconds and zero test failures — the intended
    broken-executor signal.
  • Negative controls: with the warm-up body emptied, both tests fail preemptively at 5s from
    the JUnit timeout thread (proving SEPARATE_THREAD preemption); with a 1-millisecond init
    budget, the class container fails naming warmUp() before any test runs (proving the
    init/test signal separation).
  • mvn test -pl llap-client -Dtest=TestAsyncPbRpcProxy: 2/2 green in ~0.5s (confirms surefire
    discovers the Jupiter tests).
  • Checkstyle: no new violations.

@konstantinb konstantinb changed the title HIVE-26089: ported to junit5, front-loaded JVM warup and un-ignored a test HIVE-26089: ported to junit5, front-loaded JVM warmup and un-ignored a test Aug 27, 2026
@konstantinb konstantinb changed the title HIVE-26089: ported to junit5, front-loaded JVM warmup and un-ignored a test HIVE-26089: ported TestAsyncPbRpcProxy to junit5, front-loaded JVM warmup and un-ignored a test Aug 27, 2026
@konstantinb

Copy link
Copy Markdown
Contributor Author

@abstractdog @Aggarwal-Raghav could you take a look? This supersedes the @Ignore approach of #6440 while restoring the disabled test's coverage — the description has a link to the failing precommit run, a deterministic repro, and before/after measurements under CPU starvation. Per the process note on #6440: could a committer trigger a hive-flaky-check run for this branch? Parameters: GITHUB_USER=konstantinb, GITHUB_REPO=hive, BRANCH=HIVE-26089, OPTS=-pl llap-client -Dtest=TestAsyncPbRpcProxy. (I can view the job but don't have build permission.)

@Aggarwal-Raghav Aggarwal-Raghav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @konstantinb for PR. I'm not a committer to this project so I can't trigger the flaky CI but I have given minor comments. Junit5 changes looks good.

NIT: It might be better to explicitly retain the timeout unit in milliseconds. As Hive progressively migrates to JUnit 5 (or eventually JUnit 6), keeping the exact same values?

static void warmUp() {
mock(Message.class);
mock(LlapProtocolClientProxy.ExecuteRequestCallback.class);
org.slf4j.LoggerFactory.getLogger(TestAsyncPbRpcProxy.class).info("warm-up");

@Aggarwal-Raghav Aggarwal-Raghav Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use import instead of fully pacakge classification? Also, Logger can be defined as class attribute i.e.

private static final Logger LOG = LoggerFactory.getLogger(TestAsyncPbRpcProxy.class);

@@ -32,13 +32,30 @@
import org.apache.commons.lang3.mutable.MutableInt;
import org.apache.hadoop.hive.llap.LlapNodeId;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Unused import

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants