cancel
Showing results for 
Search instead for 
Did you mean: 

Get latest file from sftp and process one at a time

hina_walia
New Contributor

I want to get all files from yesterday’s date from ftp and process them one by one. There can be only one file and can be multiple files but they need to be processed one at a time. I tried using mapper , to get yesterday value “Date.now().toLocaleDateString(‘{“format”:“yyyyMMdd”}’)”
and then use file poller, and write in file filter Labor_FDC_${ye}*.dat,
but it doesn’t work. Also at the end after load, i want to update file name in target table that it is processed ,so that it is not processed again.

17 REPLIES 17

Thanks Ajay, but still the Path variable is not getting passed in child pipeline.I have trie dusing both _Path and Path varaible in child pipelinepipeline_execute_param child_pipeline child_pipeline error with _path child_pipeline_with_path_variable

This seems to indicate that Path is not listed as a parameter in the Pipeline Properties dialog.
Can you verify that it exists?

Screen Shot 2020-03-16 at 8.52.57 AM


Diane Miller

Hello,
I have added now pipeline parameter in child pipeline as follows but still getting error. Below screenshot attached-child_pipeline_parameter_set chile_pipeline_file_redaer_error

dmiller
Former Employee

OK, in that URL, is Path a file name or a project name?


Diane Miller

Path is supposed to be file name
eg-
sftp://xxx.xx.x.xx/abc.dat