Forum Discussion

arvindnsn's avatar
arvindnsn
Contributor
2 years ago
Solved

Deleting multiple files at once from SFTP folder

Hello,   I have a process where the files needed to be deleted from the SFTP folder after the files are copied and processed.  At a given point of time there could be hundreds of files and after co...
  • darshthakkar's avatar
    darshthakkar
    9 months ago

    arvindnsn - I tried with a sample pipeline that generates 6 files over the SLDB and the performance is pretty good. It took ~1.5 mins for completion.
    I didn't connect to SFTP but just to add complexity on the same lines, I added an email snap that sent me a copy of those files, screenshots below:

     

    I have used the innate FileDelete snap for deleting multiple files at once (which does process files one by one)

    I do agree that if you're having a huge volume, this process would have to run over the weekend. Please lemme know your volume and frequency of this run to understand the issue in detail and whether we can scale up the solution.

    Thanks.

    Best Regards,

    Darsh

Ranjith's avatar
Ranjith
New Contributor II
9 months ago
Copy Link

Hi arvindnsn ,

I believe SnapLogic uses JSch SFTP libraries behind the SFTP-related snaps. In JSch (the Java SSH library), there is no direct command to delete all files in a folder without first listing them. The standard SFTP protocol, on which JSch is built, does not provide a single command to remove all files in a directory without explicitly specifying each file.

I tested a few things, such as using foldername/* with S3, but the Snap did not delete the files. I didn't get a chance to test it against SFTP, but you can give it a try by adding /* after the folder name.

If you have a subscription for the Unix Snap, you can write a shell script to connect to the SFTP and delete the files in a particular folder using the native shell command rm -rf *.