Get latest file from sftp and process one at a time

I want to get all files from yesterday’s date from ftp and process them one by one. There can be only one file and can be multiple files but they need to be processed one at a time. I tried using mapper , to get yesterday value “’{“format”:“yyyyMMdd”}’)”
and then use file poller, and write in file filter Labor_FDC_${ye}*.dat,
but it doesn’t work. Also at the end after load, i want to update file name in target table that it is processed ,so that it is not processed again.

Can some one help here? still waiting?

Hi Hina,
you can use directory browser snap to get list of file and then use sort snap to to get latest file and then use pipeline execute snap and pass the value of file name to child pipeline.

Hi Ajay,
I am getting error in pipeline execute. The _Path is not getting passed. Also, if we have multiple file, will they be processed through pipeline one at a time(latest first and so on)?
Please see attached the child pipeline which gives error
child pipeline below and child pipeline error, 3rd pic is of parent pipeline and 4th pic is pipeline execute where variable _Path is set and called in child pipeline

Also another 2 questions i have is
a) i tried to put mapper in parent pipeline and define ye as'{"format":"yyyyMMdd"}') and then tried to pass in directory "file filter" column as Labor_FDC_{ye}.dat but ye is not substituted with 202003. Please can you help here.
b)Another ques is once, file data is loaded in table, we want to enter filename in table so that it is not processed again and then delete from local folder. how can we implement this.
Thanks in advance

Hi hina,
can you please enable “=” button or expression.

and retry to validate.

in directory browser itself you can specify filter condition .


Thanks Ajay, but still the Path variable is not getting passed in child pipeline.I have trie dusing both _Path and Path varaible in child pipeline

This seems to indicate that Path is not listed as a parameter in the Pipeline Properties dialog.
Can you verify that it exists?

Screen Shot 2020-03-16 at 8.52.57 AM

I have added now pipeline parameter in child pipeline as follows but still getting error. Below screenshot attached-

OK, in that URL, is Path a file name or a project name?

Path is supposed to be file name

Provided that the account configured on the File Reader has access to the desired path location, it might be easier if someone looked at your pipeline. If you can’t post it here, you might want to have your Customer Success Manager (CSM) or Support look at it. I have alerted your CSM in case.

when i check in pipeline preview, i can see the value, but in child pipeline it looks like it is not getting passed through.

please see attached error in child pipeline

Below is screenshot parent pipeline execute shows the param preview value correctly but it is not passed in child pipeline

The child pipeline itself does not pull information from parent pipelines; parent pipelines send information to them. This allows you to reuse child pipelines with other pipelines. To test a child pipeline by itself, enter a value for your pipeline parameters.

is it possible to send me example. When i tried to hardcode value in child pipeline, it works fine but when try to get variable from parent pipeline, it throws error.Thanks a lot

If i hardcode value in child pipeline it works, but variable doesn’t work. Please see attached screenshot, first is
child pipeline set parameter and second is reading variable from file reader in child pipeline . Can you please check if syntax is fine. Thanks in advance.

Pipeline parameters only take string values, not field references.
To test this child pipeline by itself, set the Value in the Edit Pipeline dialog to the path value like sftp://xxx.xx.x.xx/abc.dat

Thanks . working fine now.