cancel
Showing results for 
Search instead for 
Did you mean: 

Too much time executing a pipeline

darshthakkar
Valued Contributor

Hi Team,

The pipeline is taking too much time to execute, isn’t that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:

image

image

CC: @dmiller @ptaylor @siwadon @koryknick

15 REPLIES 15

Thank you @koryknick for the direction.
As you can see from the snap, it took a while on the file reader and excel parser, what would you think the reason behind it and how can I avoid such things in the future?

image

image

image

Is it because the output of file reader was going in as an Input to Snowflake Execute?

koryknick
Employee
Employee

The trouble snap is the KAw&w/oAttachments - whatever that is doing, it’s the snap that is taking all of the execution time. Snaps waiting on Input and Output are generally not the issue since they aren’t really actively doing much.

That is a snowflake execute snap, it works well if there is no input view for snowflake execute however it takes a hell lot of time when the input view is enabled and something is getting ingested in that input view of snowflake execute.

koryknick
Employee
Employee

That makes sense - realize that the Execute snap would be executed once for each input document - so in your screenshots, the statement is being executed 5,850 times.

That’s quite a lot, the best development practice would be to disable the input view for the snowflake execute and use an Inner Join instead; if no output preview is getting generated then plug in the inner join later on once the dev has finished.