I’m new to SnapLogic, so apologies in advance if this is a question with an obvious answer.
We have a data lake that generates SNS notifications when different files/object types are uploaded into it, and would like to create pipelines that run in response to sets of uploads/notifications.
So, we’d like to create a parent pipeline that uses the SQS Consumer snap attached to a SQS that subscribes to the data lake SNS notifications. And, when message types “A”, “B”, and “C” are all received, it uses a Pipeline Execute snap to kick off child pipeline “X” one time. When message types “D”, “E”, and “F” are all received it uses a Pipeline Execute snap to kick off child pipeline “Y” one time, and so on.
What is the best practice way to queue up / keep track of these sets of messages across pipeline executions?