Connect SFTP server to fetch multiple files

[amarb]

Hi All,

I am trying to connect SFTP server and fetch multiple files from the source remote directory. File Reader snap has an option to get SFTP account but doesn’t find how to provide remote directory path.

In the remote directory, we are getting different types of files like *.txt where I need to apply filters based on the filenames to differentiate and invoke the different process to load into SQL DB.

Once I fetch the files from remote location how to delete the files.

Appreciate your inputs and any relevant links to look into more details. Let me know if you need any further details.

Thanks in advance,
Amar

[amarb]

Hi tstack,

I’ve tried and able to succeded below three snaps to fetch files from SFTP server

Directory Browser
File Poller
Multi File Reader

Based on suggestions tried child pipeline to get filename and file content. I tried using underscore in the File Reader but still getting same error. My goal is to extract filename to find out region from the name and extract file content to map SQL table and load into SQL server.

Appreciate your help!!

Thanks,

[tstack]

I’m attaching a project that, I think, is similar to what you want to do. You should be able to import it in the Manager and then play around with both of the pipelines it contains. The ProcessAllFiles pipeline is the parent that uses a DirectoryBrowser to look for “*.csv” files in the project. The parent pipeline then invokes the ProcessOneFile child pipeline to read it in, parse the CSV data and then format it as a JSON file that is written to the same project.

FileProcessingExample.zip (4.3 KB)

amarb:

I tried using underscore in the File Reader but still getting same error.

Are you sure the File property was marked as an expression? Based on your previous screenshot, the error mentions that it cannot find the file “…/projects-AmarnathBolleddula/$Path” which looks like it’s searching for a file with the literal name ‘$Path’ and not evaluating the expression.