Zipping whole directory with subfolders

[jfpelletier]

Hello all,

I'm having an issue with my pipeline, and would like to get some help.

The pipeline I just created simply downloads files from a server, does some transformations, then stores them temporarily directly in SnapLogic (sldb:) until the last one is downloaded. When the last file is downloaded, I have to create a ZIP package that contains all the files that I downloaded, in the same folder structure that I created.

Please note that the pipeline runs as a triggered task, and it is called when the files are ready to be downloaded individually. So I'm not downloading all the files in a single task operation, they come one at the time.

The files that I download need to be organized using the project ID as top folder, then a language code as subfolder under which I store the files for the different languages of that project.

This works fine for getting all the files in SnapLogic in the folder structure that I need. I'm trying to use the Directory Browser to list all the files, then I'm planning to write them all in the ZIP file with their respective path. My issue is that the Directory Browser returns nothing, I cannot even list the files that I just saved.

At some point I managed to make the Directory Browser to list at least the top folders, but I couldn't get the subfolders. Is the Directory Browser the correct snap to add a whole directory structure in a zip file? Is there a better technique?

Thanks in advance for any help!

Kind regards,

JF

[koryknick]

@jfpelletier - take a look at this Community post from 2021.  I believe you can get the answer to your question in those details.

[jfpelletier]

Hello @koryknick,

Thanks for this reference. I'm not sure however that it's the same issue. My issue is with the Directory Browser that is not returning the folders that I expect to see.

Originally the Directory browser was returning nothing at all, but now (I can't tell really what I changed...) it returns only the files in the project folder, but not the other subdirectories and their files. I think this is a defect, and I asked the question via a support ticket.

If I can get the list of files from the Directory browser, I should be good to zip them with the proper structure.

Thanks!

JF

[koryknick]

@jfpelletier - Then let me take a step back and say that it is not best practice to write data to the SLDB for a couple reasons.  First, you can only store files up to 100MB each, which isn't very large.  Second, anyone with Read access to your project will be able to view and download the files, which might pose a security/PII issue with production data.  Third, you probably want to remove those files from your SLDB so that it isn't accumulating over time causing issues finding the important objects stored in the SLDB.

Instead, I will recommend you look at using the pipe.tmpDir location that is a temporary storage location that is automatically removed after the pipeline execution ends (success or fail).  The size of files that can be stored there is really only limited by the temp space allocated to your snaplex execution node.  The files are transient, and not accessible to the SnapLogic UI, which removes the security concerns.

Hope this helps!

[jfpelletier]

Hello @koryknick,

Thanks a lot for your response! I fully agree about the usage of the SLDB location and I usually use the pipe.tmpDir in my pipelines for temporary files like this, but I don't think that this would work as I understand that the temp directory created lasts for the pipeline execution time only, and in my case I get the files from multiple triggered tasks, meaning that SnapLogic would delete the files as they are written to the temp folder. I plan to accumulate all the files in the SLDB until I get the last one (it might take for up to one hour to get them all), and only at that moment I want to zip the whole folder and delete it after moving out the zip file. My files are small, I'm very far from reaching any capacity limit.

By the way, I mentioned before that I suspected a defect in the Directory browser snap, and I did create a support ticket. During a call with the support agent, I reproduced the issue and showed that the Directory browser didn't find any directory on the SLDB, only files, and the support agent is now checking internally.

Kind regards,

JF