Forum Discussion

arvindnsn's avatar
arvindnsn
Contributor
2 years ago
Solved

Deleting multiple files at once from SFTP folder

Hello,

  I have a process where the files needed to be deleted from the SFTP folder after the files are copied and processed.

 At a given point of time there could be hundreds of files and after copying and processing these files, I have a separate process that deletes all these files from a SFTP folder using the "File Delete" Snap. The limitation if this snap is that it will delete one file at a time and if there are hundreds and thousands of files in the SFTP folder, it takes few mins to delete all files one by one.

I am wondering if I can use any other Snap that would delete all the files at once under a given folder. Please advise.

Thanks

Aravind N

 

 

 

  

  • darshthakkar's avatar
    darshthakkar
    10 months ago

    arvindnsn - I tried with a sample pipeline that generates 6 files over the SLDB and the performance is pretty good. It took ~1.5 mins for completion.
    I didn't connect to SFTP but just to add complexity on the same lines, I added an email snap that sent me a copy of those files, screenshots below:

     

    I have used the innate FileDelete snap for deleting multiple files at once (which does process files one by one)

    I do agree that if you're having a huge volume, this process would have to run over the weekend. Please lemme know your volume and frequency of this run to understand the issue in detail and whether we can scale up the solution.

    Thanks.

    Best Regards,

    Darsh

4 Replies

  • Ranjith's avatar
    Ranjith
    New Contributor II
    10 months ago

    Hi arvindnsn ,

    I believe SnapLogic uses JSch SFTP libraries behind the SFTP-related snaps. In JSch (the Java SSH library), there is no direct command to delete all files in a folder without first listing them. The standard SFTP protocol, on which JSch is built, does not provide a single command to remove all files in a directory without explicitly specifying each file.

    I tested a few things, such as using foldername/* with S3, but the Snap did not delete the files. I didn't get a chance to test it against SFTP, but you can give it a try by adding /* after the folder name.

    If you have a subscription for the Unix Snap, you can write a shell script to connect to the SFTP and delete the files in a particular folder using the native shell command rm -rf *. 

  • darshthakkar's avatar
    darshthakkar
    Valued Contributor
    10 months ago

    I second that Ranjith 

    arvindnsn - File Delete would be your best bet for now. Just for my references, what is the total volume you're looking at and how much time it has taken for you?
    I was thinking to process it in a batch using a pipeline execute but I'll keep you posted if I find any other workarounds.

    Best Regards,

    Darsh

  • darshthakkar's avatar
    darshthakkar
    Valued Contributor
    10 months ago

    arvindnsn - I tried with a sample pipeline that generates 6 files over the SLDB and the performance is pretty good. It took ~1.5 mins for completion.
    I didn't connect to SFTP but just to add complexity on the same lines, I added an email snap that sent me a copy of those files, screenshots below:

     

    I have used the innate FileDelete snap for deleting multiple files at once (which does process files one by one)

    I do agree that if you're having a huge volume, this process would have to run over the weekend. Please lemme know your volume and frequency of this run to understand the issue in detail and whether we can scale up the solution.

    Thanks.

    Best Regards,

    Darsh

    • arvindnsn's avatar
      arvindnsn
      Contributor
      7 months ago

      Thanks for the solution. The above one seems to be pretty effective if not super effecient especially when deleting files in thousands.