Too much time executing a pipeline

[darshthakkar]

Hi Team,

The pipeline is taking too much time to execute, isn’t that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:

CC: @dmiller @ptaylor @siwadon @koryknick

[darshthakkar]

Thank you @koryknick for the direction.
As you can see from the snap, it took a while on the file reader and excel parser, what would you think the reason behind it and how can I avoid such things in the future?

Is it because the output of file reader was going in as an Input to Snowflake Execute?

[koryknick]

The trouble snap is the KAw&w/oAttachments - whatever that is doing, it’s the snap that is taking all of the execution time. Snaps waiting on Input and Output are generally not the issue since they aren’t really actively doing much.

[darshthakkar]

That is a snowflake execute snap, it works well if there is no input view for snowflake execute however it takes a hell lot of time when the input view is enabled and something is getting ingested in that input view of snowflake execute.

[koryknick]

That makes sense - realize that the Execute snap would be executed once for each input document - so in your screenshots, the statement is being executed 5,850 times.

[darshthakkar]

koryknick:

realize that the Execute snap would be executed once for each input document - so in your screenshots, the statement is being executed 5,850 times.

That’s quite a lot, the best development practice would be to disable the input view for the snowflake execute and use an Inner Join instead; if no output preview is getting generated then plug in the inner join later on once the dev has finished.