How we can Migrate s3 Bucket folders to Another Bucket

[Rahul]

Hi,

I have a requirement to create s3 bucket folders from one bucket to another bucket with the same structure.

[bojanvelevski]

I don’t have a pipeline, but I hope this diagram can help you:

One file from each group will help you recreate the corresponding structure. Ex.: File1,File4,File7,File10

[Angel]

Hey @Rahul,

Here is different approach to solve this issue, by using Python script snap.
The below pipeline expects input document with source folder path and target folder path, and if the execution was successful it outputs document with the following properties:

• status - “successful”
• original - the input document received by the script snap

Please have a look at the example input JSON provided in JSON Generator snap.

Hope you will find that helpful.

Move-S3-Folder-Recursively.slp (9.0 KB)

[psung]

@Rahul File Operation Snap in Binary Snap pack does exactly do this. Also, it performs better because the content of each file does not go through the pipeline, rather it is coped by AWS S3 directly within AWS S3. You can set Source and Target in the Snap like s3:///sourceBucket/sourceFolder, s3:///targetBucket/targetFolder.
Select Move if you want to delete all files at the source after the file copy is completed.
One S3 account must have permission to access source and target buckets.