Get latest file from sftp and process one at a time

[hina_walia]

I want to get all files from yesterday’s date from ftp and process them one by one. There can be only one file and can be multiple files but they need to be processed one at a time. I tried using mapper , to get yesterday value “Date.now().toLocaleDateString(‘{“format”:“yyyyMMdd”}’)”
and then use file poller, and write in file filter Labor_FDC_${ye}*.dat,
but it doesn’t work. Also at the end after load, i want to update file name in target table that it is processed ,so that it is not processed again.

[hina_walia]

Can some one help here? still waiting?

[Ajay_Chawda]

Hi Hina,
you can use directory browser snap to get list of file and then use sort snap to to get latest file and then use pipeline execute snap and pass the value of file name to child pipeline.

[hina_walia]

Hi Ajay,
I am getting error in pipeline execute. The _Path is not getting passed. Also, if we have multiple file, will they be processed through pipeline one at a time(latest first and so on)?
Please see attached the child pipeline which gives error
child pipeline below and child pipeline error, 3rd pic is of parent pipeline and 4th pic is pipeline execute where variable _Path is set and called in child pipeline
!

Also another 2 questions i have is
a) i tried to put mapper in parent pipeline and define $ye as Date.now().toLocaleDateString(‘{“format”:“yyyyMMdd”}’)
and then tried to pass in directory “file filter” column as Labor_FDC_${ye}.dat but ye is not substituted with 202003. Please can you help here.
b)Another ques is once, file data is loaded in table, we want to enter filename in table so that it is not processed again and then delete from local folder. how can we implement this.
Thanks in advance

[Ajay_Chawda]

Hi hina,
can you please enable “=” button or expression.
and retry to validate.

in directory browser itself you can specify filter condition .

Regards,
Ajay