Forum Discussion
arvindnsn - I tried with a sample pipeline that generates 6 files over the SLDB and the performance is pretty good. It took ~1.5 mins for completion.
I didn't connect to SFTP but just to add complexity on the same lines, I added an email snap that sent me a copy of those files, screenshots below:
I have used the innate FileDelete snap for deleting multiple files at once (which does process files one by one)
I do agree that if you're having a huge volume, this process would have to run over the weekend. Please lemme know your volume and frequency of this run to understand the issue in detail and whether we can scale up the solution.
Thanks.
Best Regards,
Darsh
Thanks for the solution. The above one seems to be pretty effective if not super effecient especially when deleting files in thousands.