getResult(timeout) busy-loops GetWorkflowExecutionHistory in the last second before the timeout (server 1.29+) #3132

When WorkflowStub.getResult(timeout, unit, ...) (or getResultAsync) waits on a workflow that doesn't complete within the timeout, the SDK re-sends GetWorkflowExecutionHistory back to back, with no delay, for the last second before the deadline. Against a local dev server, a single 2s wait that times out sends roughly 3,000-4,000 history requests. The same code sends 1 request against server 1.28.0, so this looks like a regression that started with server 1.29.0.

With several callers waiting at the same time, the burst also hits the namespace rate limit. getResult then fails with RESOURCE_EXHAUSTED instead of throwing TimeoutException, and it returns later than the requested timeout.

Expected Behavior

getResult(2, TimeUnit.SECONDS, ...) on a workflow that is still running should wait with one or two history long polls and throw TimeoutException after 2 seconds. This is what happens against server 1.28.0 (1 request).

Actual Behavior

GetWorkflowExecutionHistory calls for a single getResult(2s) that times out on a running workflow:

• The calls are packed into the last second before the deadline, so the timeout value barely changes the count. 1s, 5s and 25s waits produced about 2,500-4,600 calls each.
• getResultAsync behaves the same way (~3,800-4,000 calls).

With concurrent waiters (server 1.32.0, N threads each calling getResult(2s) twice on the same running workflow):

The failed calls got a WorkflowServiceException caused by StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded after 2.0-2.9 seconds. In one run, a call failed with service rate limit exceeded instead. Other calls in the same namespace (DescribeWorkflowExecution and a non-waiting GetWorkflowExecutionHistory) were not throttled in these runs.

We also saw this in production on a self-hosted cluster: every short getResult wait that timed out produced a few hundred history requests. It went away after we replaced getResult with describe() and a retry from the client.

Steps to Reproduce the Problem

1. Start a dev server with Temporal CLI 1.9.1 (server 1.32.0): temporal server start-dev
2. Run the program below with Java SDK 1.40.0.
3. It prints something like getResult(2s) timed out after 2004 ms, GetWorkflowExecutionHistory calls: 2981.
4. Against a dev server from Temporal CLI 1.4.1 (server 1.28.0), the same program prints calls: 1.

import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.ClientInterceptor;
import io.grpc.MethodDescriptor;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.client.WorkflowStub;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

public class GetResultHistoryFloodRepro {

  @WorkflowInterface
  public interface SleepWorkflow {
    @WorkflowMethod
    void run();
  }

  public static class SleepWorkflowImpl implements SleepWorkflow {
    @Override
    public void run() {
      Workflow.sleep(Duration.ofMinutes(5));
    }
  }

  public static void main(String[] args) throws Exception {
    AtomicInteger historyCalls = new AtomicInteger();
    ClientInterceptor countHistoryCalls =
        new ClientInterceptor() {
          @Override
          public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
              MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
            if ("GetWorkflowExecutionHistory".equals(method.getBareMethodName())) {
              historyCalls.incrementAndGet();
            }
            return next.newCall(method, callOptions);
          }
        };

    WorkflowServiceStubs service =
        WorkflowServiceStubs.newServiceStubs(
            WorkflowServiceStubsOptions.newBuilder()
                .setTarget("127.0.0.1:7233")
                .setGrpcClientInterceptors(Collections.singletonList(countHistoryCalls))
                .build());
    WorkflowClient client = WorkflowClient.newInstance(service);
    WorkerFactory factory = WorkerFactory.newInstance(client);
    factory.newWorker("repro").registerWorkflowImplementationTypes(SleepWorkflowImpl.class);
    factory.start();

    SleepWorkflow workflow =
        client.newWorkflowStub(
            SleepWorkflow.class,
            WorkflowOptions.newBuilder()
                .setTaskQueue("repro")
                .setWorkflowId("repro-" + System.currentTimeMillis())
                .build());
    WorkflowClient.start(workflow::run);
    WorkflowStub stub = WorkflowStub.fromTyped(workflow);
    Thread.sleep(1000);

    historyCalls.set(0);
    long startedAt = System.nanoTime();
    try {
      stub.getResult(2, TimeUnit.SECONDS, Void.class);
    } catch (TimeoutException expected) {
      // The workflow sleeps for 5 minutes, so the wait times out.
    }
    System.out.printf(
        "getResult(2s) timed out after %d ms, GetWorkflowExecutionHistory calls: %d%n",
        (System.nanoTime() - startedAt) / 1_000_000, historyCalls.get());

    stub.terminate("repro done");
    factory.shutdownNow();
    service.shutdownNow();
    System.exit(0);
  }
}

Specifications

• Version: Java SDK 1.40.0 (also reproduced with 1.38.0)
• Platform: macOS (Darwin 25.4.0, arm64), OpenJDK 25.0.2
• Server: Temporal CLI dev server 1.4.1 (server 1.28.0, not affected), 1.6.1 (server 1.30.1) and 1.9.1 (server 1.32.0)
• Regression: works as expected up to server 1.28.0. Starts with 1.29.0, which includes GetWorkflowExecutionHistory long poll soft timeout temporal#8238.

Root cause

Since server 1.29.0, a GetWorkflowExecutionHistory long poll only waits until one second before the caller's deadline and then returns an empty response (temporalio/temporal#8238, "GetWorkflowExecutionHistory long poll soft timeout"). Once less than one second is left, the computed wait is zero and the server returns an empty response right away:

• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/service/history/api/get_workflow_util.go#L190-L191
• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/common/contextutil/deadline.go#L34-L36
• https://github.com/temporalio/temporal/blob/e161e3bc28d4e86ce76ae5f83d12c2443e76cb8d/common/constants.go#L36-L39

On the SDK side, WorkflowClientLongPollHelper.getInstanceCloseEvent uses the same overall deadline for every poll and sends the next request immediately after an empty response. getInstanceCloseEventAsync does the same:

• sdk-java/temporal-sdk/src/main/java/io/temporal/internal/client/WorkflowClientLongPollHelper.java

  Lines 70 to 113 in 705c3bb

  	Deadline longPollTimeoutDeadline = Deadline.after(timeout, unit);
  	while (true) {
  	GetWorkflowExecutionHistoryRequest request =
  	workflowClientHelper.newHistoryLongPollRequest(workflowExecution, pageToken);
  	try {
  	response = genericClient.longPollHistory(request, longPollTimeoutDeadline);
  	} catch (StatusRuntimeException e) {
  	if (longPollTimeoutDeadline.isExpired()
  	&& Status.Code.DEADLINE_EXCEEDED.equals(e.getStatus().getCode())) {
  	// we want to form timeout exception only if the original deadline is indeed expired.
  	// Otherwise, we should rethrow a raw DEADLINE_EXCEEDED. throwing TimeoutException
  	// in this case will be highly misleading.
  	throw newTimeoutException(workflowExecution, timeout, unit);
  	}
  	throw e;
  	}
  	History history = response.getHistory();
  	if (history.getEventsCount() > 0) {
  	HistoryEvent event = history.getEvents(0); // should be only one event
  	if (!WorkflowExecutionUtils.isWorkflowExecutionClosedEvent(event)) {
  	throw new RuntimeException("Unexpected workflow execution closing event: " + event);
  	}
  	// Workflow called continueAsNew. Start polling the new execution with new runId.
  	if (event.getEventType() == EventType.EVENT_TYPE_WORKFLOW_EXECUTION_CONTINUED_AS_NEW) {
  	pageToken = ByteString.EMPTY;
  	workflowExecution =
  	WorkflowExecution.newBuilder()
  	.setWorkflowId(workflowExecution.getWorkflowId())
  	.setRunId(
  	event
  	.getWorkflowExecutionContinuedAsNewEventAttributes()
  	.getNewExecutionRunId())
  	.build();
  	continue;
  	}
  	return event;
  	}
  	if (!response.getNextPageToken().isEmpty()) {
  	pageToken = response.getNextPageToken();
  	}
  	}

• sdk-java/temporal-sdk/src/main/java/io/temporal/internal/client/WorkflowClientLongPollAsyncHelper.java

  Lines 91 to 101 in 705c3bb

  	if (history.getEventsCount() == 0) {
  	// Empty poll returned
  	ByteString nextPageToken =
  	r.getNextPageToken().isEmpty() ? pageToken : r.getNextPageToken();
  	return getInstanceCloseEventAsync(
  	genericClient,
  	workflowClientHelper,
  	workflowExecution,
  	nextPageToken,
  	longPollTimeoutDeadline);
  	}

Together, these turn the last second into a loop of "empty response, retry right away" until the deadline. The server change was meant to reduce connection churn, but when the remaining deadline is short it multiplies the number of requests instead.

Possible fix

In the SDK, keep a minimum interval (e.g. 100ms) between polls when the previous poll came back empty, without going past the deadline. Normal long polls, which return after several seconds, are not affected.

Results from a local test with this change applied to the sync path only:

The async path needs the same rule. Since the SDK still targets Java 8, I'd like to know the preferred way to schedule the delay there (for example, reusing an existing scheduler).

Another option is to change the server so that it doesn't answer immediately when the remaining time is below the buffer. That would cover every SDK, but it's a server change.

Workaround

Don't rely on getResult(timeout) timing out while handling a request. Check the status with describe() first and call getResult only for completed executions, or let the client retry. This is what we ended up doing.

If this direction sounds reasonable, I'm happy to open a PR with the sync and async changes and a unit test. The test uses a fake client that mimics the server's soft timeout; it fails on the current code and passes with the change.