โ08-16-2022 09:31 PM
Hi Team,
The pipeline is taking too much time to execute, isnโt that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:
โ08-16-2022 10:32 PM
Pipeline execution completed after ~1.5hrs, screenshots below:
What would be the reason behind such a delay?
Regards,
Darsh
โ08-17-2022 04:40 AM
@darshthakkar - Bring up the Pipeline Execution Statistics and stay on the Snap Statistics tab. Find the snap(s) that have the longest blue bar indicating a snap that is executing most of the time. This is usually the step that is taking the longest to complete. See the image below as an example; note the two snaps where the bar is almost completely blue (File Writer and Account Read) as opposed to the other two where there is almost no total execution time (Mapper-Addingโฆ) and all green indicating that itโs waiting on input (Union):
You can also hover over the Duration (colored) bar to get the details of how much time is spent in the snap for Input, Execution, and Output.
โ08-17-2022 05:59 AM
@koryknick: Thank you for your response.
Is there a way I can go back to the snap statistics if I have run the same pipeline a couple of times after the execution Iโm wanting to investigate? I would want to understand which snaps took the highest of time as you have suggested.
As you can see from the screenshot below that even though the docs processed are low (i.e. 7392) it really took a lot of time:
I was passing the output of file reader into a snowflake execute: this might have taken a lot of time and after it took ~1.5hrs, I decided to change my design to incorporate Inner Join (wasnโt planning to use this as there is no output preview when I use Inner join even for 2000 documents selected in the output preview settings) and then this was quick, first iteration to 13m and eventually 18/19 secs (I trimmed the records to 50 thus the low time)
โ08-17-2022 06:16 AM
Yes - right there in Dashboard if you click on the โCompletedโ under the Status column, it will bring up the statistics for that execution.