Receiving Kafka Acknowledgement Time out

[pradhyumna_r]

Hi Community!! Greeting!!

We are trying to Read data from a Kafka consumer snap and are using a child pipeline (We are using Pipeline Execute) to write data into a zip folder. The pipeline is erroring with "SnapDataException: Timed out waiting for acknowledgment of consumed messages" error.
While investigating with pipeline statistics data, we have observed that the Pipeline execute snap is sending one lesser output document than the number of input documents. Example: We have 4 documents flowing from Kafka consumer. We are using Pipeline execute snap right after reading data from Kafka. Now the statistics say there are 4 input documents. But the output says 3 documents.

Contrary to this we can see 4 successful executions in the dashboard and the JCC log data confirm us with 4 triggers and 4 completion of child pipelines. This incident has been occurring intermittently. Has anyone experienced this issue? If so, please suggest us for a solution. 

Thank you!! 

[pradhyumna_r]

Hi @koryknick,@ptaylor , can you please help us with this issue. We have observed this bizarre behavior in the Pipeline Execute Snap.

[pradhyumna_r]

The error is we can see is on the Kafka Consumer saying "Timed out waiting for acknowledgment of consumed messages"

[ptaylor]

Hello,

Is the Pipe Exec configured to execute all children in the same local node? The Acknowledge snap must execute on the same node as the Consumer. So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.

To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.

[pradhyumna_r]

Hi @ptaylor, Please find our response to the queries shared.
Q: Is the Pipe Exec configured to execute all children in the same local node?
A: Yes they are.

Q: The Acknowledge snap must execute on the same node as the Consumer.So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.
A: Yes they both are in snap parent pipeline. - Child pipeline is configured to run on LOCAL_NODE

Q: To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.
A: Yes, Auto Commit is disabled, and we have also configured "Wait after each batch of records".

Awaiting for your response, Thanks!