Skip to content

Flaky integration test: TaskClientTests.testSearchTasks() times out against indexed search #145

Description

@ambiorix2099

Summary

TaskClientTests.testSearchTasks() intermittently fails with a SocketTimeoutException against the live server, failing the Java Client Integration Tests workflow on PRs that do not touch any related code.

Most recent occurrence: run 30394076277, triggered by #144 — a documentation-only PR whose diff cannot reach this code path (tests depends only on :conductor-client; the PR's sole Java edit is a comment in conductor-client-ai, which that module never compiles).

TaskClientTests > testSearchTasks() FAILED
    com.netflix.conductor.client.exception.ConductorClientException: timeout}
        at com.netflix.conductor.client.http.TaskClient.search(TaskClient.java:555)
        at io.orkes.conductor.client.http.TaskClientTests.testSearchTasks(TaskClientTests.java:740)
    Caused by:
    java.net.SocketTimeoutException: timeout
        at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:675)

106 tests completed, 1 failed, 4 skipped

Failure duration was 35.573s — the server did not return response headers within the client read timeout. No assertion in the test ever evaluated.

Root cause

The test starts a workflow, sleeps a fixed interval, then queries the eventually-consistent indexed search endpoint with no retry or polling:

String workflowId = workflowClient.startWorkflow(request);
try {
    Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
    var result = taskClient.search("workflowId='" + workflowId + "'");
    assertNotNull(result);
} finally {
    workflowClient.terminateWorkflow(workflowId, "cleanup");
}

A fixed sleep against an async index is inherently timing-dependent. The file's history already contains a commit titled more wait time (f020a53), which increased the sleep rather than removing the dependence on it — so this has been papered over before.

Suggested fix

Replace the fixed sleep with bounded polling, so the test waits only as long as needed and fails with a meaningful message rather than a socket timeout:

  • Poll taskClient.search(...) until it returns a non-empty result or a timeout budget expires (Awaitility, or a simple bounded retry loop consistent with whatever this suite already uses elsewhere).
  • Treat SocketTimeoutException as retryable within the budget, since a slow index response is the observed failure mode rather than a wrong result.

Impact

Unrelated PRs get a red Integration Tests check, which trains reviewers to ignore the signal. Worth fixing in its own PR where a live-server test change can be reviewed on its merits.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions