Forum Discussion

amarb's avatar
amarb
New Contributor II
8 years ago

Connect SFTP server to fetch multiple files

Hi All, I am trying to connect SFTP server and fetch multiple files from the source remote directory. File Reader snap has an option to get SFTP account but doesn’t find how to provide remote direc...
pipeline
chenry's avatar
chenry
Former Employee
8 years ago

Hi Amar,
Sorry for the delay. I was thinking more about your scenario, and I think I have a better suggestion. Since you may not need to “poll” for the files, I am thinking a different Snap would be better.

Say I want to boil the problem down to grabbing the files, and then deleting them…You could use a MultiFileReader Snap, with a file filter to get the desired types of file. The output will be a binary data stream.

You could connect this to a BinaryToDocument Snap and then perform operations on that document. At some point you would need to use a Mapper to key on the filename and pass that to a FileDelete Snap.

For the simpler case of just say…reading files and deleting them for proof of concept, you could use a DirectoryBrowser instead of the MultiFileReader. The DirectoryBrowser would return you attributes about all of the files that match the type filter (essentially a List in document format versus binary). Then you’d plug this into a Mapper Snap, to map the filename. Then you’d plug this into a Delete Snap where you configure it to read the file name from the mapped variable.

I hope this helps you get further towards your goal!

Thanks,
-Charlie

  • amarb's avatar
    amarb
    New Contributor II
    8 years ago

    Hi Charlie,

    Thank you, for the giving other options. I am still struggling to process the files from the first snap.
    I am providing my findings for your reference. Appreciate your inputs further.

    MultiFileReader Snap is not taking sftp path under folder/file section. I am getting below exception:
    The same path is working other Snaps but not in MultiFileReader Snap. Provide your suggestions based on the error.

    Multi File Reader( Multi File Reader )
    Reads binary data from an endpoint (e.g. file, sldb, s3, sftp, ftp, adl, etc).

    Snap has the following errostrong textrs. Please open the info box and edit the highlighted fields.
    Could not compile expression: _sftp://ftp.ext.hpe.com:2222/_SC … (Reason: Encountered extraneous input ‘:’ at line 1:4; expecting={, ‘||’, ‘&&’, ‘^’, ‘==’, ‘!=’, ‘>’, ‘<’, ‘>=’, ‘<=’, ‘+’, ‘-’, ‘*’, ‘/’, ‘%’, ‘?’, ‘instanceof’, ‘in’, ‘[’, ‘(’, ‘.’, PropertyRef}; Resolution: Please check expression syntax)

    I’ve tried DirectoryBrowser and able to fetch files from the SFTP server and used mapper snap to take filename into the variable. If I use file delete snap again it is throwing same error as multi-file reader snap exception. The file parameter is not taking SFTP path with variable filename.

    Appreciate your further support and also let me know how to extract data from the original file. The attributes from the directory browser snap showing bytes and name only.

    Thanks in advance,
    Amar

    • chenry's avatar
      chenry
      Former Employee
      8 years ago

      Hi Amar,
      Let’s first start with the Multi File Reader Snap. If you create a new one, and then in the account tab, create a new account. For one of our internal test accounts, I chose basic auth. I am prompted for user name password, which I give. Next in the main settings tab of the Snap, I input the following under file/folder the format: sftp://hostName/DirectoryStructure/TargetFolder/
      Then in the wildcard I put “.csv” And I am able to run successfully. If I look at my output as JSON, I can see “content-location” which includes the absolute path to the file name (including the file name). From this point, you should be able to use a BinaryToDocument Snap in order to operate on the document data. But let’s see how things go up to this point first.

      Thanks, and apologies for the delay.
      -Charlie

      • amarb's avatar
        amarb
        New Contributor II
        8 years ago

        Thank you, Charlie. I’ve followed your steps for the Multi File Reader Snap and able to succeed to fetch files from the remote directory.
        Attached screenshot for your ref.

        I also used BinaryToDocument snap and tried all options under encode or decode parameters. But I am getting below message in the dialogue window.

        “Preview data is not found or too large for browser to decrypt”

        Appreciate your next steps to get the document data.

        Thanks,
        Amar