Repository navigation
Retry known service errors and 5xx responses - #955
Conversation
| error_registry=error_registry, | ||
| context=context, | ||
| ) | ||
| raise self._apply_status_retry_info(error, response) |
There was a problem hiding this comment.
A transient 5xx without the Smithy-Protocol header (e.g. a 503 from a load balancer that omits it) doesn't reach this: the header check raises first, so it stays is_retry_safe=None. Can we run that error through _apply_status_retry_info too?
smithy-python/packages/smithy-http/src/smithy_http/protocols.py
Lines 750 to 759 in bb2c9f7
bb2c9f7 to
4f4a41a
Compare
| fault=fault, | ||
| is_throttling_error=is_throttle, | ||
| is_timeout_error=is_timeout, | ||
| is_retry_safe=is_throttle or is_timeout or None, |
There was a problem hiding this comment.
This was the right time and place to determine retry eligibility.
The protocol is the primary HTTP-aware component, and hands off this HTTP-based retryability judgement to the HTTP-unaware retry strategy.
We only needed to add 5xx and network errors to the retry criteria.
There was a problem hiding this comment.
To fill out what was discussed internally:
With smithy-http holding protocols like RPCv2, which also need to be awsQueryCompat, smithy-http now has AWS concerns mixed in.
The correct thing to do may have been to import and subclass an AWS version of RPCv2 within smithy-aws-core. This means smithy-aws-core is acting as extension for both smithy-core and smithy-http. Optionally, we create smithy-aws-http to mirror smithy-http.
So, regardless of whether we create "smithy-aws-http", smithy-http should serve its protocols by having a retry-eligibility or "service error classification" system.
It should perform the HTTP status code and network error mapping:
def is_retryable_error(http_response: HttpResponse, other_args...) -> bool:
...I mention other_args because retry eligibility is not entirely decided by the HTTP response. The absence of HTTP communication, for example due to a network outage, is also retryable.
So then, whatever is in the AWS downstream, whether smithy-aws-core or smithy-aws-http (hypothetical), should define its own additional criteria for AWS-specific requirements like specific error names and error codes from specific services.
def is_retryable_aws_error(http_response: HttpResponse, other_args...) -> bool:
return is_retryable_error(...) or additional_conditionsWith these two functions in place, smithy (without AWS) protocols call the former, and Smithy protocols (operating within AWS clients) or AWS protocols can call the latter.
There was a problem hiding this comment.
One note: The function needs to also mark throttling and timeout errors using the same known list. To centralize that, the function won't return a boolean and instead marks the necessary fields on the error for retryability and the error category, then returns it.
4f4a41a to
428a9d4
Compare
428a9d4 to
d59d784
Compare
| return status == 408 and not modeled | ||
|
|
||
|
|
||
| def _is_transient_error(status: int, code: str | None) -> bool: |
There was a problem hiding this comment.
how are failures to connect handled now? network disconnection should be considered transient, but does not involve a status code number
There was a problem hiding this comment.
This should happen at the transport layer. Each transport defines a TIMEOUT_EXCEPTIONS list to wrap the underlying transport's errors as a retryable ClientTimeoutError. We can do a similar approach for network connection errors with a CONNECTION_EXCEPTIONS and a retryable ClientConnectionError.
There was a problem hiding this comment.
Will open a new PR to wrap connection errors as retryable in the transport layer.
Description
Updates runtime to retry errors that services don't mark as
@retryable, such as EC2'sRequestLimitExceeded.Protocols now mark these errors as retryable:
Throttling,ThrottlingException,ThrottledException,RequestThrottledException,TooManyRequestsException,ProvisionedThroughputExceededException,TransactionInProgressException,RequestLimitExceeded,BandwidthLimitExceeded,LimitExceededException,RequestThrottled,SlowDown,PriorRequestNotComplete,EC2ThrottledExceptionRequestTimeout,InternalError, andRequestTimeoutExceptionFor query-compatible services, the code in the
x-amzn-query-errorheader is the one matched.Errors modeled as not retryable are never changed. The two kinds of rule combine, so a 503
SlowDownis retried as a throttling error.Note
STS's
IDPCommunicationErrorretry customization was previously included in this PR. However, with the context property removed, we can no longer check the error code in an interceptor.AssumeRoleWithWebIdentityis the only operation that modelsIDPCommunicationError. Thus, when other STS operations return it, they become unmodeled since we only have per-operation error registries. An interceptor won't be able to just check that the error matches the modeled type. We need to either 1) use service-wide type registries for a fallback or 2) expose the error code onCallError, response metadata, or the context property like before. This will be a follow up that doesn't block this PR.Other changes
Testing
ec2queryclient.make test-unit- 4596 passed, 7 skipped.make test-functionalandmake build-javapass.aws-sdk-python. Their functional and integration tests pass, and these previously skipped functional tests now pass:test_retry_reuses_the_checksumBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.