We still see that there are quite a lot of integrations where we depend on files to be sent and picked up.
I find no good solution of triggering tasks by detecting files in a folder other than to build one or many pipelines that runs frequently with directory browser that check files from a folder and then route the data to the pipeline that will handle the transformation/execution.
And even if i set a schedule interval to run every minute, it is not very reliable, as you can see on the start time the interval is quite far from every minute .
This type of setup will quickly fill up the dashboard with a lot of executions, especially if we need to poll frequently.
It would be awesome if there was a way to trigger a pipeline if a file is added in a folder…
With support for the common protocols that we have in the file snaps and (reliable) customizable intervals for polling, this simply need to send the same type of information to the pipeline as we have from a directory browser snap.
I would be happy to elaborate on a suggested setup but not sure if i’m the only one with this “problem”.