Connect SFTP server to fetch multiple files
Hi All,
I am trying to connect SFTP server and fetch multiple files from the source remote directory. File Reader snap has an option to get SFTP account but doesn’t find how to provide remote direc...
Forum Discussion
The error is due to the child pipeline failing. I would suggest debugging the child pipeline on its own before trying to run it from the parent. If you haven’t already, add a ‘Path’ pipeline parameter to the child. Note that referencing a pipeline parameter is referenced in an expression using an underscore instead of a dollar sign. So, your FileReader snap should use ‘_Path’ for the File property. When defining the pipeline parameter, you can give a default value that is a test file path so that the child pipeline will run correctly. Then, when the child is executed from the parent pipeline, the default value will be replaced with the ‘Path’ passed in via the PipelineExecute snap.
Hi tstack,
I’ve tried and able to succeded below three snaps to fetch files from SFTP server
Directory Browser
File Poller
Multi File Reader
Based on suggestions tried child pipeline to get filename and file content. I tried using underscore in the File Reader but still getting same error. My goal is to extract filename to find out region from the name and extract file content to map SQL table and load into SQL server.
Appreciate your help!!
Thanks,
I’m attaching a project that, I think, is similar to what you want to do. You should be able to import it in the Manager and then play around with both of the pipelines it contains. The ProcessAllFiles pipeline is the parent that uses a DirectoryBrowser to look for “*.csv” files in the project. The parent pipeline then invokes the ProcessOneFile child pipeline to read it in, parse the CSV data and then format it as a JSON file that is written to the same project.
FileProcessingExample.zip (4.3 KB)
Are you sure the File property was marked as an expression? Based on your previous screenshot, the error mentions that it cannot find the file “…/projects-AmarnathBolleddula/$Path” which looks like it’s searching for a file with the literal name ‘$Path’ and not evaluating the expression.
