How we can Migrate s3 Bucket folders to Another Bucket
Hi,
I have a requirement to create s3 bucket folders from one bucket to another bucket with the same structure.
Forum Discussion
7 Replies
@Rahul File Operation Snap in Binary Snap pack does exactly do this. Also, it performs better because the content of each file does not go through the pipeline, rather it is coped by AWS S3 directly within AWS S3. You can set Source and Target in the Snap like s3:///sourceBucket/sourceFolder, s3:///targetBucket/targetFolder.
Select Move if you want to delete all files at the source after the file copy is completed.
One S3 account must have permission to access source and target buckets.Hey @Rahul,
You can easily read one file that in the current bucket directories, and re-write it into the new bucket by replacing the old bucket with the new one in the content-location header.
Make sure you check the Create directory if not present option. You may achieve this with File Operation as well, but I’m not sure.
@bojanvelevski Thanks But in Source s3 folder have multiple subfolders and subfolders have multiple subfolders. we need to follow same hierarchy and create in Target S3 Bucket folder.
Yes, I already assumed that. My suggestion was to get all folders and subfolders with multiple Directory Browsers and at the end, one file from each lowest level is enough to recreate the whole structure.
I don’t have a pipeline, but I hope this diagram can help you:
One file from each group will help you recreate the corresponding structure. Ex.: File1,File4,File7,File10
Hey @Rahul,
Here is different approach to solve this issue, by using Python script snap.
The below pipeline expects input document with source folder path and target folder path, and if the execution was successful it outputs document with the following properties:- status - “successful”
- original - the input document received by the script snap
Please have a look at the example input JSON provided in JSON Generator snap.
Hope you will find that helpful.
Move-S3-Folder-Recursively.slp (9.0 KB)
