Receiving Kafka Acknowledgement Time out

[pradhyumna_r]

Hi Community!! Greeting!!

We are trying to Read data from a Kafka consumer snap and are using a child pipeline (We are using Pipeline Execute) to write data into a zip folder. The pipeline is erroring with "SnapDataException: Timed out waiting for acknowledgment of consumed messages" error.
While investigating with pipeline statistics data, we have observed that the Pipeline execute snap is sending one lesser output document than the number of input documents. Example: We have 4 documents flowing from Kafka consumer. We are using Pipeline execute snap right after reading data from Kafka. Now the statistics say there are 4 input documents. But the output says 3 documents.

Contrary to this we can see 4 successful executions in the dashboard and the JCC log data confirm us with 4 triggers and 4 completion of child pipelines. This incident has been occurring intermittently. Has anyone experienced this issue? If so, please suggest us for a solution. 

Thank you!! 

[ptaylor]

Please send me a link to a pipeline execution which shows this issue.

[koryknick]

@pradhyumna_r - The Kafka Consumer snap will only output one message at a time, which must be confirmed before the next message is sent.  I would check your child pipeline to ensure that it is returning all data back to the parent.  For example, is there a filter or an error in the child that prevents the message from returning back?

[pradhyumna_r]

Hi @koryknick Yes, we have configured "Wait after each batch of records". And please find our child pipeline configuration snip. As we are not filtering for any objects but are flagging the pipeline for Retry on Write failure to destination path. 

[koryknick]

@pradhyumna_r - please check the Dashboard (or Monitor) and look at the execution statistics on the child pipeline for one of your jobs that have failed.  My guess is that you have an unexpected error that is not returning the data back to the parent.  

If not, then please provide your configuration of the Pipeline Execute snap.  One question here - are you intending that each message off of the Kafka queue is written to it's own zipfile?

Or you could open a support ticket for assistance if you feel that the behavior is incorrect.

[pradhyumna_r]

1. The Dashboard indicates all the child pipelines have completed successfully and doesnt have any errors. 

Q : If not, then please provide your configuration of the Pipeline Execute snap. One question here - are you intending that each message off of the Kafka queue is written to it's own zipfile?
A: Yes. We are reading and splitting. 
Please find our Config for Pipeline Execute below,