09-29-2021 03:30 AM
I have a pipeline which reads data from an S3 location and writes to an SMB location with no transformations. The pipeline involves 2 main snaps, S3 reader connected to Zip write snap. The Zip write snap recreates the whole structure of the S3 location in the Zip file, which I would like to avoid and place all the files in either a root directory or a directory of my choice instead of copying the S3 structure.
As there is no transformation, I do not want to parse the files and parse back again. Is there a way of doing this?
09-29-2021 12:51 PM
Hi @SuryaReddy,
I cannot understand the reason why you have to write Zip file , and read it after? Why not a plain S3 read, and write to SMB? Are you using the same path from the S3, to set the file path to SMB? I’m just guessing. Example:
s3:///mybucket@s3.eu-test-1.amazonaws.com/test_files/csv/input.csv => smb://smb.Snaplogic.com:445/test_files/csv/input.csv
If this is the case, you can easily take the filename only, and define the target path on your own, if not, please send some more details.
09-30-2021 01:17 AM
Hi @bojanvelevski ,
The pipeline does this:
Read files (and there are multiple files to read) from S3 bucket ==> Write to an SMB Location as a zipped file.
When doing this, the zip file has the same folder structure as the S3, which is very deep. I am looking to avoid this deep structure and make a much more flat structure.
Example:
S3 file folder structure: Bucket → Folder1 → Folder2 → Folder3 → File
Zip file structure: Zip file name (User defined) → Bucket → Folder1 → Folder2 → Folder3 → File
Required: Zip file name (User defined) → Bucket → Folder1 → File
09-30-2021 03:01 AM
That’s because you are passing the content location as a target filepath. Use the following expression to get the filename only:
$['content-location'].split('/')[$['content-location'].split('/').length -1]
than, you can define the path as you like it to be. Example:
"smb://smb.Snaplogic.com:445/Bucket/Folder1/" + $['content-location'].split('/')[$['content-location'].split('/').length -1]
Don’t forget to enable the expressions functionality.
10-01-2021 09:36 AM
Test pipeline 0_2021_10_01.slp (3.0 KB)
I have uploaded a stripped down version of the pipeline, can you advise how to change the path you mention above?