I have a pipeline that starts with a BigQuery Execute (BQE) Snap, followed by several other Snaps.
I only want the step(s) following the BQE Snap to execute if that Snap returns data i.e. if no rows are returned, do nothing, otherwise continue with the rest of the pipeline
Figured it out - I can use a Data Validator Snap and set the constraint for one of the column names i.e. Required = true
Figured it out - I can use a Data Validator Snap and set the constraint for one of the column names i.e. Required = true
With the Data Validator Snap (and the constraint that you've put), if no rows are returned, the snap (Data validator) will throw an error and the pipeline will fail. - or maybe that's the behavior that you want to achieve. 🙂
Another option that you can check is Filter Snap. If the filter expression evaluates to true, the document is passed through. If the filter expression evaluates to false, the document is dropped in the stream.
BR,
Spiro
Yes, sorry could've made that clearer - I only want the pipeline to execute if there is data, because the second phase sends a Slack notification, and I only want that notification sent if the data is present (error logs).
I like the filter recommendation - thank you!
