Design Approach for Downloading SLDB saved files using smb

Hello, looking for input regarding best approach to automate the manual download of various types of files (.csv, .xlsx, .zip) which are saved in the same project folder.
We want to keep them available in Snaplogic but also download to another server on certain days of the month.
File Operation /Copy does not seem like an option.
Thanks!

What about Copy and a File Writer? At the same time you can write the file to SLDB and to another server, or use a File Reader/File Writer if you want to keep it as a separate process.

1 Like

@SusanB

Do you want to automate the process of coping the files from SLDB to another server? Or, do you want to copy files from one source location/server to SLDB and target server?

If first, then one pipeline with Directory Browser Snap to scan and get files from SLDB, File Reader Snap(to read the files) and File Writer Snap(to write files to target location). Also you can create a scheduled task to run on predestined time.

If second, probably you will have two pipelines:
1) One pipeline that will read files from source server to SLDB.
2) Second pipeline that will read files from SLDB to target server.

Also, those pipelines can be scheduled to run based on your requirements.

File Operation Snap will work only with: local file (file:///), FTP, SFTP, S3, WASB, and WASBS - file protocols.

Regards,
Spiro Taleski

Thanks. I did see that we would have to (re)read and then use File Writer. This may be an option with a copy/router to only execute on the desired day(s) of month.

Thank you for the options/insight.
I think we’re probably looking at the first scenario. The files are generated daily and saved in the Snaplogic folder (SLDB?).
On certain days of the month they need to be copied to a target server.

I think whatever approach we decide on, we’ll need to (re)read the files based on their file types and then use the corresponding file writer snap to write to the target server.

Appreciate everyone’s input.

1 Like

@SusanB

Having a scheduled process/task that will run on certain days of the month, scanning the SLDB project/directory for a files(using Directory Browser Snap), File Reader Snap(to read the files), and File Writer Snap(to write the file to target system), will work, but also you should think of a logic where once copied files from SLDB to not be picked again in the next execution.

Regards,
Spiro Taleski

Hi Spiro,
Thanks for your additional input. Luckily the filenames already include a timestamp.
Knowing this, the new pipeline will include a parameter of the list of month days that need to have files copied. IE “1,15” (for the first and fifteenth of each month).
The pipe will have a router that compares today’s month date to the parameter list. If today is not in the parameter’s list of values we exit. If it is we perform the copy steps:
Find and filter the list of files that contain today’s date, route them for zip vs non zip files, read and write (adding the option to Overwrite and/or Ignore if the file is already in the target directory. Currently zip file is having issue with ignore - so we’ll just overwrite those).
We will then schedule the task to run daily.
When needed for special requests we can easily add another temporary scheduled task for other days.
As time allows, I may attempt to make this more flexible by adding a parameter for specific file name prefixes.
For now this should eliminate the need for a human to manually copy files.

1 Like