New spring-batch-s3 module#182
Conversation
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
andreacioni
force-pushed
the
spring-batch-s3
branch
from
b6e9643 to
e3f3ce1
Compare
dgray16
left a comment
dgray16
left a comment
There was a problem hiding this comment.
Thanks for your work!
Here are some places that could be improved from my point of view.
...tch-s3/src/main/java/org/springframework/batch/extensions/s3/stream/S3MultipartUploader.java
Show resolved
Hide resolved
spring-batch-s3/src/test/java/org/springframework/batch/extensions/s3/S3ItemReaderTests.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/springframework/batch/extensions/s3/stream/S3MultipartOutputStreamTests.java
Outdated
Show resolved
Hide resolved
...3/src/test/java/org/springframework/batch/extensions/s3/stream/S3MultipartUploaderTests.java
Outdated
Show resolved
Hide resolved
...tch-s3/src/test/java/org/springframework/batch/extensions/s3/stream/S3OutputStreamTests.java
Outdated
Show resolved
Hide resolved
scordio
left a comment
scordio
left a comment
There was a problem hiding this comment.
Hi @andreacioni, thanks for the PR!
I suggest adding a CI job, see for example the one for Spring Batch Notion.
Also, the new module should be mentioned in the root README.
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
- Builders moved to a dedicated package - project README.adoc updated; - root README.md updated: add reference to new module; - use org.springframework.util.unit.DataSize to express data sizes; - removed apache-client exclusion in pom.xml to allow user to not specify an actual implementation in their dependencies - updated spring-javaformat-checkstyle to match the latest version - update gitignore to match the one of Spring Batch Notion - Add Maven Wrapper executables - Add Github Action workflow - Removed maven-wrapper.jar - Code cleanup Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
- pom.xml reformatted - root README missing Amazon S3 link - atLeastOnce abandoned in favor of times(x) - minor corrections Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
dgray16
left a comment
dgray16
left a comment
There was a problem hiding this comment.
It would be nice to have a review from @fmbenhassine also.
...ch-s3/src/main/java/org/springframework/batch/extensions/s3/builder/S3ItemWriterBuilder.java
Outdated
Show resolved
Hide resolved
spring-batch-s3/src/main/java/org/springframework/batch/extensions/s3/stream/Defaults.java
Outdated
Show resolved
Hide resolved
spring-batch-s3/src/test/java/org/springframework/batch/extensions/s3/S3ItemReaderTests.java
Show resolved
Hide resolved
Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
…ions/s3/S3ItemReaderTests.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
…ions/s3/builder/S3ItemWriterBuilder.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
…ions/s3/stream/Defaults.java Co-authored-by: Volodymyr <dgray16@users.noreply.github.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
|
Any news on this? I don't think I'm alone in wanting this in. |
|
Hi @andreacioni Thank you for contributing this extension! I did not follow the SDK updates from AWS (the last time I checked was when I wrote spring-projects/spring-batch#3818 (comment)), but I have no objection to add this S3 extension if they now provide the necessary bits to properly create an item reader and writer for Spring Batch. However, we need someone to lead and maintain this module after merging this PR. If you are interested, please reach out to me by email (my email is the one I use in git) to explain the responsibility of extension leads and the onboarding process to the Best regards, |
scordio
left a comment
scordio
left a comment
There was a problem hiding this comment.
Hi @andreacioni, I have a few follow-up comments, but feel free to ignore them until things are sorted out with @fmbenhassine.
| * memory. | ||
| */ | ||
|
|
||
| package org.springframework.batch.extensions.s3; |
There was a problem hiding this comment.
With the migration to Batch 6 and Boot 4, I'd also consider the Jspecify integration.
For this, you might take inspiration from what I've done at a9648a7.
There was a problem hiding this comment.
Thanks for the review @scordio , this is the hardest issue as I never heard about Jspecify 😅. I'll submit a commit in the next days to solve this.
There was a problem hiding this comment.
No worries! I'm also happy to contribute to your branch, if you are okay with it.
There was a problem hiding this comment.
Hello @scordio, I should have pushed all the Jspecify adaptation and the build gets completed. However I'm not sure how to test it because I've tried to put some null on the tests but there is no error in the compile phase. Is this the supposed behavior?
There was a problem hiding this comment.
Let me take a closer look at your branch in the next few days. I'll get back to you 🙂
Co-authored-by: Stefano Cordio <stefano.cordio@gmail.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Co-authored-by: Stefano Cordio <stefano.cordio@gmail.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Co-authored-by: Stefano Cordio <stefano.cordio@gmail.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Co-authored-by: Stefano Cordio <stefano.cordio@gmail.com> Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
|
Hello @fmbenhassine, glad to hear you are interested in this. I tried to reach you by email (_at_broadcom.com) in October 22 but I haven't heard anything from you, please let me know which are the formal steps to be involved in this extension maintenance. |
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Indeed, It was on November 22nd and I completely missed it (it was in the spam folder..). Apologies for that. I will reply by email and send you the invitation on GitHub. Best regards, |
...-s3/src/main/java/org/springframework/batch/extensions/s3/serializer/S3StringSerializer.java
Outdated
Show resolved
Hide resolved
...ng-batch-s3/src/main/java/org/springframework/batch/extensions/s3/stream/S3OutputStream.java
Show resolved
Hide resolved
|
This is a very nice contribution, well done @andreacioni, thank you for sharing it! I added a couple minor comments, but overall, this is awesome 👌 @dgray16 @scordio Thank you for following up on this and for you reviews, much appreciated 🙏 I love how this community around Spring Batch extensions is growing and to see folks helping each other, amazing! |
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
andreacioni
force-pushed
the
spring-batch-s3
branch
from
007ef50 to
47fb00f
Compare
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
andreacioni
force-pushed
the
spring-batch-s3
branch
from
47fb00f to
dd100a5
Compare
Signed-off-by: Andrea Cioni <supercionci@hotmail.it>
5 participants
Hello everyone,
after struggling a bit with S3 and Spring Batch I want to contribute to this ecosystem with my little knowledge by proposing this PR to your attention. In the past I've followed some discussion that lead me here (mainly #3818). I had to work with S3 in the past years and I had to implement such Reader/Writer each time, moreover it's a new addition to AWS SDK the API to upload objects with unknown size (reference) from streams. Given this inputs I decided that was time for Spring Batch to have some support for S3, hope this will be welcome.
More information are in the README, please let me know if there is something missing I'd be happy to fix it.