Receiving Kafka Acknowledgement Time out
Hi Community!! Greeting!! We are trying to Read data from a Kafka consumer snap and are using a child pipeline (We are using Pipeline Execute) to write data into a zip folder. The pipeline is erro...
Forum Discussion
Hello,
Is the Pipe Exec configured to execute all children in the same local node? The Acknowledge snap must execute on the same node as the Consumer. So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.
To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.
Hi ptaylor, Please find our response to the queries shared.
Q: Is the Pipe Exec configured to execute all children in the same local node?
A: Yes they are.Q: The Acknowledge snap must execute on the same node as the Consumer.So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.
A: Yes they both are in snap parent pipeline. - Child pipeline is configured to run on LOCAL_NODEQ: To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.
A: Yes, Auto Commit is disabled, and we have also configured "Wait after each batch of records".
Awaiting for your response, Thanks!Please send me a link to a pipeline execution which shows this issue.
