Forum Discussion

hina_walia's avatar
hina_walia
New Contributor
6 years ago

Get latest file from sftp and process one at a time

I want to get all files from yesterday’s date from ftp and process them one by one. There can be only one file and can be multiple files but they need to be processed one at a time. I tried using mapper , to get yesterday value “Date.now().toLocaleDateString(‘{“format”:“yyyyMMdd”}’)”
and then use file poller, and write in file filter Labor_FDC_${ye}*.dat,
but it doesn’t work. Also at the end after load, i want to update file name in target table that it is processed ,so that it is not processed again.

17 Replies

    • Ajay_Chawda's avatar
      Ajay_Chawda
      Contributor
      6 years ago

      Hi Hina,
      you can use directory browser snap to get list of file and then use sort snap to to get latest file and then use pipeline execute snap and pass the value of file name to child pipeline.

      • hina_walia's avatar
        hina_walia
        New Contributor
        6 years ago

        Hi Ajay,
        I am getting error in pipeline execute. The _Path is not getting passed. Also, if we have multiple file, will they be processed through pipeline one at a time(latest first and so on)?
        Please see attached the child pipeline which gives error
        child pipeline below and child pipeline error, 3rd pic is of parent pipeline and 4th pic is pipeline execute where variable _Path is set and called in child pipeline
        !

        Also another 2 questions i have is
        a) i tried to put mapper in parent pipeline and define $ye as Date.now().toLocaleDateString(‘{“format”:“yyyyMMdd”}’)
        and then tried to pass in directory “file filter” column as Labor_FDC_${ye}.dat but ye is not substituted with 202003. Please can you help here.
        b)Another ques is once, file data is loaded in table, we want to enter filename in table so that it is not processed again and then delete from local folder. how can we implement this.
        Thanks in advance

  • dmiller's avatar
    dmiller
    Former Employee
    6 years ago

    OK, in that URL, is Path a file name or a project name?

    • hina_walia's avatar
      hina_walia
      New Contributor
      6 years ago

      Path is supposed to be file name
      eg-
      sftp://xxx.xx.x.xx/abc.dat

      • dmiller's avatar
        dmiller
        Former Employee
        6 years ago

        Provided that the account configured on the File Reader has access to the desired path location, it might be easier if someone looked at your pipeline. If you can’t post it here, you might want to have your Customer Success Manager (CSM) or Support look at it. I have alerted your CSM in case.

  • dmiller's avatar
    dmiller
    Former Employee
    6 years ago

    Pipeline parameters only take string values, not field references.
    To test this child pipeline by itself, set the Value in the Edit Pipeline dialog to the path value like sftp://xxx.xx.x.xx/abc.dat