08-16-2022 09:31 PM
Hi Team,
The pipeline is taking too much time to execute, isn’t that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:
08-17-2022 08:17 AM
Thank you @koryknick for the direction.
As you can see from the snap, it took a while on the file reader and excel parser, what would you think the reason behind it and how can I avoid such things in the future?
Is it because the output of file reader was going in as an Input to Snowflake Execute?
08-17-2022 08:51 AM
The trouble snap is the KAw&w/oAttachments - whatever that is doing, it’s the snap that is taking all of the execution time. Snaps waiting on Input and Output are generally not the issue since they aren’t really actively doing much.
08-17-2022 10:29 AM
That is a snowflake execute snap, it works well if there is no input view for snowflake execute however it takes a hell lot of time when the input view is enabled and something is getting ingested in that input view of snowflake execute.
08-17-2022 10:36 AM
That makes sense - realize that the Execute snap would be executed once for each input document - so in your screenshots, the statement is being executed 5,850 times.
08-17-2022 11:47 AM
That’s quite a lot, the best development practice would be to disable the input view for the snowflake execute and use an Inner Join instead; if no output preview is getting generated then plug in the inner join later on once the dev has finished.