feat: project setup and upload file #1712

0marperez · 2025-10-17T13:55:48Z

Issue #

N/A

Description of changes

Project setup
Publishing config
Transfer manager client
Business metric
Transfer interceptors
Upload file operation
- Concurrent uploads
- MPU part buffering
- Code generated IO
- Code generated type converters

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-10-17T14:02:55Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:14:53Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:42:33Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:53:02Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T15:11:43Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

lauzadis

Nice start

...mmon/src/aws/sdk/kotlin/runtime/http/interceptors/businessmetrics/AwsBusinessMetricsUtils.kt

hll/build.gradle.kts

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/jvm/test/aws/sdk/kotlin/hll/s3transfermanager/UploadFileTest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

...ransfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/BusinessMetricInterceptor.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/TransferInterceptor.kt

hll/s3-transfer-manager/build.gradle.kts

github-actions · 2025-10-21T14:07:26Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-30T03:03:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T04:09:45Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T15:51:50Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T16:13:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

gradle/libs.versions.toml

...n/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/codegen/operations/rendering/OperationRenderer.kt

hll/s3-transfer-manager-codegen/build.gradle.kts

...odegen/src/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/MappingTypes.kt

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

...ger/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/HelperFunctions.kt

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/InitiateTransfer.kt

...common/test/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/UploadObjectTest.kt

github-actions · 2025-10-30T20:40:09Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

hll/s3-transfer-manager-codegen/build.gradle.kts

settings.gradle.kts

gradle/libs.versions.toml

ianbotsf · 2025-10-31T22:58:52Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


Comment: 👍 Yes, this is in the right place. We don't want to publish this since we have no use case for it right now. We can scratch the TODO.

This doesn't disable publication right? Just API validation and docgen

This is configured in repo tools right?

hll/hll-codegen/build.gradle.kts

ianbotsf · 2025-11-03T18:16:46Z

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

+    val mpuUploadId = initiateTransfer(
+        multipartUpload,
+        transferContext,
+        contentLength,
+        uploadFileRequest,
+        interceptors,
+        client,
+    )
+
+    val uploadedParts = transferBytes(
+        multipartUpload,
+        contentLength,
+        partSizeBytes,
+        logger,
+        uploadFileRequest,
+        transferContext,
+        mpuUploadId,
+        interceptors,
+        client,
+        maxInMemoryParts,
+        maxConcurrentPartUploads,
+    )
+
+    completeTransfer(
+        multipartUpload,
+        transferContext,
+        uploadFileRequest,
+        mpuUploadId,
+        uploadedParts,
+        interceptors,
+        client,
+    )


Style: The volume of arguments passed into these functions is too great. Can any of these be grouped into objects, derived from other parameters, etc.? This level of data coupling is an indicator that we might be better served modelling a base operation type which can be implemented for each operation type (e.g., UploadFile) or for each subtype (e.g., UploadFileSingle and UploadFileMultipart), which would reduce the amount of if (multipartUpload) calls.

I left a TODO for this, once we write other operations it'll be easier to create a common abstraction.

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

ianbotsf · 2025-11-03T18:32:36Z

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/phases/TransferBytes.kt

+) = produce(
+    capacity = maxInMemoryParts,
+) {


Correctness: Isn't maxInMemoryParts supposed to limit the parts for the entire S3TM? This looks like it applies to individual objects but we'll be parallelizing multi-object transfers.

I decided to use a Channel for each object but while keeping track of total parts in memory using a Semaphore.

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

lauzadis · 2025-11-03T19:13:26Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


This doesn't disable publication right? Just API validation and docgen

lauzadis · 2025-11-03T19:19:16Z

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

+internal val uploadFileConversions = listOf(
+    ConversionMapping(
+        source = TypeRef(
+            "aws.sdk.kotlin.services.s3.model",
+            "PutObjectResponse",
+        ),
+        destination = TypeRef(
+            "aws.sdk.kotlin.hll.s3transfermanager.model",
+            "UploadFileResponse",
+        ),
+        setOf(
+            "bucketKeyEnabled",
+            "checksumCrc32",
+            "checksumCrc32C",
+            "checksumCrc64Nvme",
+            "checksumSha1",
+            "checksumSha256",
+            "checksumType",
+            "eTag",
+            "expiration",


I don't think a data string will make this any more readable / maintainable? The internal spec already has this modeled as a JSON list, we can make changes (which I believe are unlikely) by inspecting the diff of that file

hll/s3-transfer-manager/build.gradle.kts

lauzadis · 2025-11-03T19:27:25Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+    /**
+     * Preferred part size for multipart uploads.
+     * If using this size would require more than 10,000 parts (the S3 limit),
+     * the smallest possible part size that results in 10,000 parts is used instead.
+     *
+     * Default to 8,000,000 bytes.
+     */
+    public val partSizeBytes: Long = builder.partSizeBytes


This is my fault since I left a comment asking to change it to just partSize to simplify the name. We are logging a warning when deviating from the configured part size. It's not a strong opinion so I will let @0marperez make the decision

#1712

github-actions · 2025-11-07T00:13:18Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

ianbotsf

Looking pretty good, down to the final sticking point!

ianbotsf · 2025-11-11T16:43:33Z

...codegen/src/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/renderers/IoRenderer.kt

+                write("@PublishedApi")
+                write(
+                    "internal fun build(): #1L = #1L(this)",
+                    className,
+                )


Question: Why is @PublishedApi necessary here? Is this getting inlined somewhere?

It's getting inlined by the public invoke function. It used to be public but you asked to make it internal. I used @PublishedApi because it's what we use often in the rest of the codebase.

...tlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/utils/S3TransferManagerCodegenException.kt

ianbotsf · 2025-11-11T17:02:21Z

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/phases/TransferBytes.kt

+            executePhase(
+                TransferPhase.BytesTransferred,
+                localContext,
+                interceptors,
+            ) {
+                localContext.s3Response = client.withTmBusinessMetric {
+                    it.uploadPart(localContext.s3Request as UploadPartRequest)
+                }
+
+                // -1 part in memory
+                semaphore.release()
+
+                localContext.transferredBytes = localContext.transferredBytes!! + partSize
+            }


Question: executePhase only invokes the given block if no exception is thrown first, right? When an exception is thrown, what should happen to the part in memory and the semaphore? Presently it looks like they stay stuck and won't be resolved.

I added a try/catch block around the consumer to make sure the parts don't stay in memory and we update the semaphore.

I'm guessing your question is for when users are try/catching their S3 TM requests and then continue making S3 TM calls?

ianbotsf · 2025-11-11T17:05:43Z

...sfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/interceptors/TransferContext.kt

+/**
+ * The context around an [aws.sdk.kotlin.hll.s3transfermanager.S3TransferManager] transfer.
+ * Used to track transfer progress or to modify in progress transfers.
+ */
+public data class TransferContext(
+    override val s3Request: Any? = null,
+    override val s3Response: Any? = null,
+    override val tmRequest: Any? = null,
+    override val tmResponse: Any? = null,
+    override val transferableBytes: Long? = null,
+    override val transferredBytes: Long? = null,
+    override val transferableObjects: Long? = null,
+    override val transferredObjects: Long? = null,
+) : TransferInterceptorContext
+
+/**
+ * The context around an [aws.sdk.kotlin.hll.s3transfermanager.S3TransferManager] transfer.
+ * Used to track transfer progress or to modify in progress transfers.
+ */
+public data class MutableTransferContext(
+    override var s3Request: Any? = null,
+    override var s3Response: Any? = null,
+    override var tmRequest: Any? = null,
+    override var tmResponse: Any? = null,
+    override var transferableBytes: Long? = null,
+    override var transferredBytes: Long? = null,
+    override var transferableObjects: Long? = null,
+    override var transferredObjects: Long? = null,
+) : TransferInterceptorContext {
+    internal fun immutableCopy() =
+        TransferContext(
+            s3Request,
+            s3Response,
+            tmRequest,
+            tmResponse,
+            transferableBytes,
+            transferredBytes,
+            transferableObjects,
+            transferredObjects,
+        )
+}


Correctness: We must not expose data class types in our public API. Can the implementations be internal instead?

Changed these to classes. I think they need to be public or either internal with @PublishedApi since they're exposed to the user in TransferInterceptor.

...r/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/HelperFunctions.kt

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/S3TransferManagerException.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

github-actions · 2025-11-12T19:37:58Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

0marperez added 6 commits October 6, 2025 17:39

feat: s3 transfer manager client

bc2fa0a

saving work - upload file

6895fc3

self review

c9486f1

pull from main

22243e7

misc: merge from main

dbff057

finish upload file

835866a

provide static dummy credentials to tests

8edf604

0marperez added the no-changelog Indicates that a changelog entry isn't required for a pull request. Use sparingly. label Oct 17, 2025

add s3 as a test dependency?

4137873

refactor tests to be JVM only

d84516d

Depend on missing 'DefaultAwsSigner'

9d5ec07

0marperez marked this pull request as ready for review October 17, 2025 15:14

0marperez requested a review from a team as a code owner October 17, 2025 15:14

lauzadis requested changes Oct 17, 2025

View reviewed changes

Merge branch 'main' of github.com:aws/aws-sdk-kotlin into s3-tm-dev

71ea5b7

ianbotsf requested changes Oct 20, 2025

View reviewed changes

feedback

fb2417f

0marperez mentioned this pull request Oct 30, 2025

feat: sdk source readRemaining smithy-lang/smithy-kotlin#1454

Open

feedback v2

a193a2f

add missing KDocs and self review

010bd4b

lauzadis mentioned this pull request Oct 30, 2025

S3 Transfer Manager #972

Open

1 task

remove old TODO

8a821d6

latest feedback

ac6c88e

lauzadis reviewed Oct 30, 2025

View reviewed changes

feedback v3

32aae24

ianbotsf requested changes Nov 3, 2025

View reviewed changes

lauzadis approved these changes Nov 3, 2025

View reviewed changes

feedback round 3?

ac24d5a

ianbotsf requested changes Nov 11, 2025

View reviewed changes

more feedback

4950835

0marperez requested a review from ianbotsf November 12, 2025 19:31


		"s3-transfer-manager-codegen", // TODO: Disable publishing ?

feat: project setup and upload file #1712

Are you sure you want to change the base?

feat: project setup and upload file #1712

Uh oh!

Conversation

0marperez commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #

Description of changes

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

lauzadis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

0marperez commented Oct 17, 2025 •

edited

Loading