Move endpoint resolution from interceptors to pipeline stage#6820
Move endpoint resolution from interceptors to pipeline stage#6820S-Saranya1 wants to merge 6 commits intofeature/master/core-interceptors-migrationfrom
Conversation
87d0960 to
305ba3a
Compare
6000d9f to
7b09efe
Compare
…l/endpoint-resolution-refactoring
| } | ||
|
|
||
| // Apply resolved endpoint URL | ||
| ClientEndpointProvider clientEndpointProvider = |
There was a problem hiding this comment.
I see that in the design doc, there's a backward compatibility issue and is solved by adding a check for customer modified url here. Why is that missed here?
codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/ClientClassUtils.java
Show resolved
Hide resolved
| return executionAttributes.getAttribute(AwsExecutionAttribute.FIPS_ENDPOINT_ENABLED); | ||
| } | ||
|
|
||
| /** |
There was a problem hiding this comment.
Just curious why do we deleted the comments here and below?
There was a problem hiding this comment.
+1, I think the comments provide some value, it's not super obvious what's happening
| "TestValue0")))) | ||
| .hasMessageContaining("stop"); | ||
|
|
||
| assertThat(interceptor.context.httpRequest().matchingHeaders("TestHeader")).containsExactly("TestValue", "TestValue0"); |
There was a problem hiding this comment.
This is a subtle behavioral change and we can call it out as a known behavior change in the feature branch merging PR description. if downstream system is doing exact string matching on header value ordering (which is wrong) this could break them.
There was a problem hiding this comment.
Will call this out in feature branch merging PR
There was a problem hiding this comment.
Ditto here, we need codegen tests
There was a problem hiding this comment.
There are several codegen fixture tests to update, will create a separate PR for those to keep this one focused. For now, updated a few test files and added to this PR so the codegen changes are easy to review, also added the new codegen fixture test EndpointResolverUtilsSpecTest.
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package software.amazon.awssdk.awscore.internal.endpoints; |
There was a problem hiding this comment.
Is this being used across modules? If so, it shouldn't be internal
There was a problem hiding this comment.
Yeah right, but the callers are also @SdkInternalApi right, so kept it that way, but WDYT should we leave it here or move it to a non-internal package.
| @@ -38,10 +52,6 @@ public static Boolean fipsEnabledBuiltIn(ExecutionAttributes executionAttributes | |||
| return executionAttributes.getAttribute(AwsExecutionAttribute.FIPS_ENDPOINT_ENABLED); | |||
| } | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
+1, I think the comments provide some value, it's not super obvious what's happening
| } | ||
|
|
||
| EndpointResolver resolver = attrs.getAttribute(SdkInternalExecutionAttribute.ENDPOINT_RESOLVER); | ||
| if (resolver == null) { |
There was a problem hiding this comment.
Is there a valid case where there's no resolver?
There was a problem hiding this comment.
Hmm no valid case, removed that check.
| ClassName awsEndpointAttribute = ClassName.get("software.amazon.awssdk.awscore.endpoints", "AwsEndpointAttribute"); | ||
| ClassName endpointAuthScheme = ClassName.get("software.amazon.awssdk.awscore.endpoints.authscheme", "EndpointAuthScheme"); |
There was a problem hiding this comment.
Is this necessary? codegen has a dependency on aws-core so you should be able to reference the class directly
codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/specs/JsonProtocolSpec.java
Show resolved
Hide resolved
| /** | ||
| * Generates a per-service endpoint resolver utility class (e.g. {@code DynamoDbEndpointResolverUtils}) | ||
| * containing all static helper methods for endpoint resolution: building endpoint params, host prefix | ||
| * resolution, auth scheme property copying, and business metrics. |
There was a problem hiding this comment.
Do we need to code generate this at all? What service specific logic does it have vs what is generic? If its generic, I'd lean towards moving it out of our codegeneration and into a core package. That makes codegen simpler, reduces duplication between services (reducing artifact size) and makes it easier to unit test.
There was a problem hiding this comment.
Hmm yeah, this needs to be code-generated. Most of the logic is building rule params, extracting context params, and applying host prefixes which is service-specific. These methods reference per-service types like S3EndpointParams, GetObjectRequest, etc. that live in each service module, so a shared class in core can't reference them.
There was a problem hiding this comment.
Ah right, I got myself confused by EndpointResolverUtil and AwsEndpointProviderUtils face palm.
How difficult would it be to add the service prefix to this name, as we do for things like S3EndpointParams and S3EndpointProvider?
...rc/main/java/software/amazon/awssdk/awscore/internal/endpoints/AwsEndpointProviderUtils.java
Show resolved
Hide resolved
core/sdk-core/src/main/java/software/amazon/awssdk/core/spi/endpoint/EndpointResolver.java
Outdated
Show resolved
Hide resolved
| * no way to know, just from the HTTP request object, where the override path ends (if it's even there) and where | ||
| * the request path starts. Additionally, the rule itself may also append other parts to the endpoint override path. | ||
| * <p> | ||
| * To solve this issue, we pass in the endpoint set on the path, which allows us to the strip the path from the |
There was a problem hiding this comment.
Assume this was pre-existing, but I think wording is incorrect:
we pass in the endpoint set on the path
I think this should be "we pass in the endpoint set on the client" (ie, path -> client)
| /** | ||
| * Generates a per-service endpoint resolver utility class (e.g. {@code DynamoDbEndpointResolverUtils}) | ||
| * containing all static helper methods for endpoint resolution: building endpoint params, host prefix | ||
| * resolution, auth scheme property copying, and business metrics. |
There was a problem hiding this comment.
Ah right, I got myself confused by EndpointResolverUtil and AwsEndpointProviderUtils face palm.
How difficult would it be to add the service prefix to this name, as we do for things like S3EndpointParams and S3EndpointProvider?
| * Detects if a customer interceptor modified the HTTP request URL in modifyHttpRequest(). | ||
| * Compares the current request's host and scheme against the snapshot taken before interceptors ran. | ||
| */ | ||
| private static boolean customerModifiedUrl(SdkHttpFullRequest.Builder request, ExecutionAttributes attrs) { |
There was a problem hiding this comment.
naming nit, maybe interceptorModifiedEndpoint instead of customer?
There was a problem hiding this comment.
Ah right, I got myself confused by EndpointResolverUtil and AwsEndpointProviderUtils face palm.
How difficult would it be to add the service prefix to this name, as we do for things like S3EndpointParams and S3EndpointProvider?
Yeah while generating it generates as DynamoDbEndpointResolverUtils,S3EndpointResolverUtils
| } | ||
|
|
||
| /** | ||
| * Detects if a customer interceptor modified the HTTP request URL in modifyHttpRequest(). |
There was a problem hiding this comment.
Just to be clear - so, we don't care if an interceptor modified the path right? That will be accounted for in the merge. Its only if they have changed the schema/host/port, ect that we would care right?
There was a problem hiding this comment.
Correct, we only check host, scheme. For path, we're keeping it as is since setUri() handles the merging. We only skip endpoint application if the host, scheme changed.
| if (preModifyUri == null) { | ||
| return false; | ||
| } | ||
| String requestHost = request.host(); |
There was a problem hiding this comment.
Do we need to also check port? (probalby an edge case, but not sure)? Any other components that might matter?
There was a problem hiding this comment.
Yeah good call, added port check. Don't think there are other components that matter.
| ClientEndpointProvider clientEndpointProvider = | ||
| attrs.getAttribute(SdkInternalExecutionAttribute.CLIENT_ENDPOINT_PROVIDER); | ||
| if (customerModifiedUrl(request, attrs)) { | ||
| return request; |
There was a problem hiding this comment.
I can't remember exactly what our logic was previously here.... but is there a case (like S3 Path style) where we still need to apply parts of the resolved endpoint (but keep the interceptor modified host/schema ect)? I'm not 100% sure about this, just trying to think through some possible edge cases here.
Motivation and Context
The SDK uses ExecutionInterceptor for core endpoint resolution functionality. But this interface was
designed for customer extensibility, not internal SDK. This is Part 2 of the core interceptors migration, following the auth scheme resolution refactoring in #6755.
Modifications
1. Codegen — generate resolveEndpoint() in client classes
2. Codegen — generate per-service EndpointResolverUtils
3. Added EndpointResolutionStage
New pipeline stage in sdk-core that runs after AuthSchemeResolutionStage. Calls the service-specific callback, stores RESOLVED_ENDPOINT, copies endpoint headers, applies resolved URL, and reports ENDPOINT_RESOLVE_DURATION metric. Registered in both sync and async pipelines.
4. Added EndpointResolver interface and ClientExecutionParams
EndpointResolver functional interface in sdk-core, endpointResolver field on ClientExecutionParams, ENDPOINT_RESOLVER execution attribute, and wiring in AwsExecutionContextBuilder.
5. De-duplicated AwsEndpointProviderUtils
Created single shared class in aws-core replacing identical per-service generated copies. Deleted the AwsEndpointProviderUtils.java.resource template.
6. Updated S3 presigner and S3Utilities
Removed old endpoint interceptors from both. Added direct endpoint resolution methods that call S3EndpointResolverUtils.ruleParams() and apply the resolved URL.
7. Deleted dead code
Removed EndpointResolverInterceptorSpec, RequestEndpointInterceptorSpec, AuthSchemeInterceptorSpec, their tests, AwsEndpointProviderUtils.java.resource template, and dead generateAuthSchemeInterceptor() method.
Testing
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License