Conditional Snap Run
Hi everyone,
I have been experiencing an issue lately where I have a large zip file containing 17 CSV files. I unzip the file and write it into an external stage. After that, I run several procedures on Snowflake using Snowflake Multi Execute.
My concern is that SnapLogic tends to unzip File A, write File A into the external stage, and then run the Snowflake Multi Execute Snap, next to unzip File B etc.. What I want is for the Snowflake Multi Execute Snap to only run after the File Writer Snap has successfully finished writing all the 17 files into the stage.
I have tried a workaround where I use a Router Snap to count the output documents from the File Writer Snap. If the count is less than 17, I wait. If it is equal to or greater than 17, I run the following snaps. This workaround works well, but it is not the optimal solution for me, especially if tomorrow the zip file contains 18, 19, or 20 files (the pipeline will automatically fail)
Do you have a better way to design this pipeline?
Thank you.