Throttle Pipeline based on a combination threshold

[jpanikulam]

Hi,

We have a need to trigger a child pipeline when an input SQS message count reaches ‘N’ messages OR ‘M’ seconds have passed since the last run - whichever occurs earlier. Any thoughts on how best to write this pipeline?

The goal is to try and throttle these fast incoming messages and process in groups rather than one by one since otherwise can result in a lot of processing and network traffic.

Thanks,
Joseph

[aleung]

jpanikulam:

put SQS message count reach

This probably need a bit more details to resolve. Feel free to reach out to me.

[karansawhney]

Hello @aleung, I have a similar use case. Can we talk?