SQS Consumer snap is ready all message at pipeline startup

[vish_ch]

Hi,
I have certain observations while running a pipeline with SQS snap as trigger.
1. On startup, pipeline reads all available messages from sqs and hold them in buffer and process them as per concurrency
2. There is no mechanism to control the speed at which we poll messages from queue.

If you have used sqs consumer snap in your pipelines, how did you tackle above issues.
Any throttling mechanism did you implement to control this.


Thanks,
Vishal.

[alchemiz]

Set a value in "Message count" property of the SQS Consumer snap to control the number of messages to be consumed in each trigger

As for the speed, it depends if you're doing client-acknowledge (synchronous) or auto-acknowledge (asynchronous)

If Auto acknowledge is unchecked, you need to add an SQS Acknowledge snap to acknowledge the message, once message has been acknowledge only then the SQS will stream the next message if available.

[vish_ch]

With message_count, it only ready defined messages and stops which means we need to run our pipeline at defined schedule.
We need to run this on real-time basis hence not the good option for us