Skip to content

Conversation

@Dunedan
Copy link

@Dunedan Dunedan commented Jul 4, 2017

dynamodb-replicator uses / as delimiter to hierarchically structure
the data in S3. Up to now it did a naive join of all path parts when
constructing paths and key names for S3. That produced different paths
in S3 whether or not a trailing slash was provided when specifying the
target. For example the following two commands produced different output
files in S3:

incremental-backfill us-east-1/table s3://s3-bucket/foo
incremental-backfill us-east-1/table s3://s3-bucket/foo/

As that's different from other tools which interact with S3, like the
official awscli this commit changes that behavior to work like
path.join does, but by using an external package to ensure that also
under Windows paths are generated with forward slashes.

Daniel Roschka added 2 commits July 4, 2017 13:58
`dynamodb-replicator` uses `/` as delimiter to hierarchically structure
the data in S3. Up to now it did a naive join of all path parts when
constructing paths and key names for S3. That produced different paths
in S3 whether or not a trailing slash was provided when specifying the
target. For example the following two commands produced different output
files in S3:

    incremental-backfill us-east-1/table s3://s3-bucket/foo
    incremental-backfill us-east-1/table s3://s3-bucket/foo/

As that's different from other tools which interact with S3, like the
official `awscli` this commit changes that behavior to work like
`path.join` does, but by using an external package to ensure that also
under Windows paths are generated with forward slashes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant