Skip to content

Retry known service errors and 5xx responses - #955

Merged
arandito merged 1 commit into
developfrom
retry-known-errors
Nov 19, 2026
Merged

arandito merged 1 commit into
developfrom
retry-known-errors

Conversation

@arandito

@arandito arandito commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Updates runtime to retry errors that services don't mark as @retryable, such as EC2's RequestLimitExceeded.

Protocols now mark these errors as retryable:

  • 500, 502, 503, and 504 responses as transient errors, whether or not the error is modeled.
  • Unmodeled 429 responses as throttling errors and unmodeled 408 responses as timeouts. The other protocols already did this and now RPCv2 protocols do too.
  • Throttling - Throttling, ThrottlingException, ThrottledException, RequestThrottledException, TooManyRequestsException, ProvisionedThroughputExceededException, TransactionInProgressException, RequestLimitExceeded, BandwidthLimitExceeded, LimitExceededException, RequestThrottled, SlowDown, PriorRequestNotComplete, EC2ThrottledException
  • Transient - RequestTimeout, InternalError, and RequestTimeoutException

For query-compatible services, the code in the x-amzn-query-error header is the one matched.

Errors modeled as not retryable are never changed. The two kinds of rule combine, so a 503 SlowDown is retried as a throttling error.

Note

STS's IDPCommunicationError retry customization was previously included in this PR. However, with the context property removed, we can no longer check the error code in an interceptor. AssumeRoleWithWebIdentity is the only operation that models IDPCommunicationError. Thus, when other STS operations return it, they become unmodeled since we only have per-operation error registries. An interceptor won't be able to just check that the error matches the modeled type. We need to either 1) use service-wide type registries for a fallback or 2) expose the error code on CallError, response metadata, or the context property like before. This will be a follow up that doesn't block this PR.

Other changes

  • Fixes - JSON error bodies without an error type are now unmodeled errors instead of parse failures.

Testing

  • Added unit tests for the status and error code rules across all protocols, and end-to-end retry tests through the generated ec2query client.
  • Ran make test-unit - 4596 passed, 7 skipped. make test-functional and make build-java pass.
  • Regenerated all clients in aws-sdk-python. Their functional and integration tests pass, and these previously skipped functional tests now pass:
    • S3 test_retry_reuses_the_checksum

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

error_registry=error_registry,
context=context,
)
raise self._apply_status_retry_info(error, response)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A transient 5xx without the Smithy-Protocol header (e.g. a 503 from a load balancer that omits it) doesn't reach this: the header check raises first, so it stays is_retry_safe=None. Can we run that error through _apply_status_retry_info too?

if actual_protocol != self._smithy_protocol:
yield from self._discard_body(operation, response, read_body)
raise CallError(
message=(
f"Malformed response for operation {operation.schema.id}: "
f"expected Smithy-Protocol header {self._smithy_protocol!r}, "
f"got {actual_protocol!r} - status: {response.status}"
),
fault="client" if response.status < 500 else "server",
)

@arandito
arandito marked this pull request as ready for review November 18, 2026 00:50
@arandito
arandito requested a review from a team as a code owner November 18, 2026 00:50
fault=fault,
is_throttling_error=is_throttle,
is_timeout_error=is_timeout,
is_retry_safe=is_throttle or is_timeout or None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the right time and place to determine retry eligibility.

The protocol is the primary HTTP-aware component, and hands off this HTTP-based retryability judgement to the HTTP-unaware retry strategy.

We only needed to add 5xx and network errors to the retry criteria.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fill out what was discussed internally:

With smithy-http holding protocols like RPCv2, which also need to be awsQueryCompat, smithy-http now has AWS concerns mixed in.

The correct thing to do may have been to import and subclass an AWS version of RPCv2 within smithy-aws-core. This means smithy-aws-core is acting as extension for both smithy-core and smithy-http. Optionally, we create smithy-aws-http to mirror smithy-http.

So, regardless of whether we create "smithy-aws-http", smithy-http should serve its protocols by having a retry-eligibility or "service error classification" system.

It should perform the HTTP status code and network error mapping:

def is_retryable_error(http_response: HttpResponse, other_args...) -> bool:
    ...

I mention other_args because retry eligibility is not entirely decided by the HTTP response. The absence of HTTP communication, for example due to a network outage, is also retryable.

So then, whatever is in the AWS downstream, whether smithy-aws-core or smithy-aws-http (hypothetical), should define its own additional criteria for AWS-specific requirements like specific error names and error codes from specific services.

def is_retryable_aws_error(http_response: HttpResponse, other_args...) -> bool:
   return is_retryable_error(...) or additional_conditions

With these two functions in place, smithy (without AWS) protocols call the former, and Smithy protocols (operating within AWS clients) or AWS protocols can call the latter.

@arandito arandito Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One note: The function needs to also mark throttling and timeout errors using the same known list. To centralize that, the function won't return a boolean and instead marks the necessary fields on the error for retryability and the error category, then returns it.

return status == 408 and not modeled


def _is_transient_error(status: int, code: str | None) -> bool:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are failures to connect handled now? network disconnection should be considered transient, but does not involve a status code number

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should happen at the transport layer. Each transport defines a TIMEOUT_EXCEPTIONS list to wrap the underlying transport's errors as a retryable ClientTimeoutError. We can do a similar approach for network connection errors with a CONNECTION_EXCEPTIONS and a retryable ClientConnectionError.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will open a new PR to wrap connection errors as retryable in the transport layer.

@arandito
arandito merged commit 0064dac into develop Nov 19, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants