Too much time executing a pipeline
Hi Team,
The pipeline is taking too much time to execute, isn’t that strange when it takes around 40+ minutes on a single file reader snap? Snaps below for reference:
CC: @dmiller @pt...
Forum Discussion
If you really want performance out of this, I would recommend that you load the ID values into another Snowflake table and run your complex query using that table of IDs as your filtering condition in the query itself. The load of IDs to that table can be done using the Snowflake Bulk Load snap and should be pretty quick - then the query using the Snowflake Execute only has to run once to return all data.
Agreed, this makes a lot of sense but currently we don’t have our bulk snaps setup in the Org. In the interim, I believe Inner Join will do the job as there is a paradigm shift in the performance. Really appreciate your suggestions though.
