Connect SFTP server to fetch multiple files

[amarb]

Hi All,

I am trying to connect SFTP server and fetch multiple files from the source remote directory. File Reader snap has an option to get SFTP account but doesn’t find how to provide remote directory path.

In the remote directory, we are getting different types of files like *.txt where I need to apply filters based on the filenames to differentiate and invoke the different process to load into SQL DB.

Once I fetch the files from remote location how to delete the files.

Appreciate your inputs and any relevant links to look into more details. Let me know if you need any further details.

Thanks in advance,
Amar

[amarb]

Hi Charlie,

I’ve tried Directory Browser snap to fetch files. When I hook pipelineExecute Snap I am getting the error.

Failure: Pipeline did not complete successfully, Reason: Snap errors: {ruuid=830fa0d3-2253-40b5-b419-2a4fd53dfa97, reason=Value referenced in the sub-expression ‘$Path’ is null, label=File Reader, resolution=Please check expression syntax and data types.}, Resolution: Fix the child pipeline errors and try again

I might be giving the wrong expression in the parameter section. Can you please look into pipeline Execute and File reader snap screenshots and suggest the correction.

Attached screenshots.

Thanks,
Amar

[robin]

@amarb, the “Execution Label” setting is not expression-enabled. Please enable it by selecting the “=” button beside the field. As it is not enabled, it is treating the $Path value as literal.

[del]

robin:

@amarb, the “Execution Label” setting is not expression-enabled. Please enable it by selecting the “=” button beside the field. As it is not enabled, it is treating the $Path value as literal.

Also, the Pipeline Parameters section doesn’t look right to me. Typically, the Parameter Name won’t be prefixed with a ‘$’ (although yours might). As well, the Parameter Value should be enabled for expressions by toggling the “=” button.

[amarb]

Thank you Robin and Del. I’ve tried both ways and still getting an error. Attached screenshots for your ref.
Is there anything I need to provide in the child pipeline. File Reader snap I am using as shown in the previous screenshot.

[tstack]

amarb:

I’ve tried both ways and still getting an error.

The error is due to the child pipeline failing. I would suggest debugging the child pipeline on its own before trying to run it from the parent. If you haven’t already, add a ‘Path’ pipeline parameter to the child. Note that referencing a pipeline parameter is referenced in an expression using an underscore instead of a dollar sign. So, your FileReader snap should use ‘_Path’ for the File property. When defining the pipeline parameter, you can give a default value that is a test file path so that the child pipeline will run correctly. Then, when the child is executed from the parent pipeline, the default value will be replaced with the ‘Path’ passed in via the PipelineExecute snap.