Too much time executing a pipeline
Hi Team,
The pipeline is taking too much time to execute, isn’t that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:
CC: @dmiller @pt...
Forum Discussion
If using Gate, make sure you change the “Output Document Content” from “All Input Documents” to either of the other options to prevent everything from going to memory. You could also use the Tail snap if you don’t need to send the input documents to the Snowflake Execute snap.
I don’t know what you’re trying to do in this pipeline, but removing the input view from the Snowflake Execute would have it execute as soon as the pipeline starts, which might not be the behavior you want.
Yes, good points, Kory, especially about disabling the input view.
Tail would probably make more sense than Gate if there’s only one input view. I couldn’t recall the name of that option in the Gate snap (even though I added it!) so just said “and note its options”. 🙂
