Zipping whole directory with subfolders
Hello all, I'm having an issue with my pipeline, and would like to get some help. The pipeline I just created simply downloads files from a server, does some transformations, then stores them tempo...
Forum Discussion
jfpelletier - Then let me take a step back and say that it is not best practice to write data to the SLDB for a couple reasons. First, you can only store files up to 100MB each, which isn't very large. Second, anyone with Read access to your project will be able to view and download the files, which might pose a security/PII issue with production data. Third, you probably want to remove those files from your SLDB so that it isn't accumulating over time causing issues finding the important objects stored in the SLDB.
Instead, I will recommend you look at using the pipe.tmpDir location that is a temporary storage location that is automatically removed after the pipeline execution ends (success or fail). The size of files that can be stored there is really only limited by the temp space allocated to your snaplex execution node. The files are transient, and not accessible to the SnapLogic UI, which removes the security concerns.
Hope this helps!
Hello koryknick,
Thanks a lot for your response! I fully agree about the usage of the SLDB location and I usually use the pipe.tmpDir in my pipelines for temporary files like this, but I don't think that this would work as I understand that the temp directory created lasts for the pipeline execution time only, and in my case I get the files from multiple triggered tasks, meaning that SnapLogic would delete the files as they are written to the temp folder. I plan to accumulate all the files in the SLDB until I get the last one (it might take for up to one hour to get them all), and only at that moment I want to zip the whole folder and delete it after moving out the zip file. My files are small, I'm very far from reaching any capacity limit.
By the way, I mentioned before that I suspected a defect in the Directory browser snap, and I did create a support ticket. During a call with the support agent, I reproduced the issue and showed that the Directory browser didn't find any directory on the SLDB, only files, and the support agent is now checking internally.
Kind regards,
JF
