05-20-2024 05:18 AM
Hi Community!! Greeting!!
We are trying to Read data from a Kafka consumer snap and are using a child pipeline (We are using Pipeline Execute) to write data into a zip folder. The pipeline is erroring with "SnapDataException: Timed out waiting for acknowledgment of consumed messages" error.
While investigating with pipeline statistics data, we have observed that the Pipeline execute snap is sending one lesser output document than the number of input documents. Example: We have 4 documents flowing from Kafka consumer. We are using Pipeline execute snap right after reading data from Kafka. Now the statistics say there are 4 input documents. But the output says 3 documents.
Contrary to this we can see 4 successful executions in the dashboard and the JCC log data confirm us with 4 triggers and 4 completion of child pipelines. This incident has been occurring intermittently. Has anyone experienced this issue? If so, please suggest us for a solution.
Thank you!!
06-16-2024 11:45 PM
Hi @koryknick, please find our answers in the above reply and help us with your insights on it.
Thanks!
06-21-2024 07:10 AM
@pradhyumna_r - I believe this is just an issue of the document being filtered in your child pipeline execution and the parent not able to acknowledge it. Since I can't see the details in your provided screenshots, I recommend that you open a support ticket through your company's SnapLogic contacts on this issue for further assistance.