Forum Discussion

pradhyumna_r's avatar
pradhyumna_r
New Contributor
2 years ago

Receiving Kafka Acknowledgement Time out

Hi Community!! Greeting!!

We are trying to Read data from a Kafka consumer snap and are using a child pipeline (We are using Pipeline Execute) to write data into a zip folder. The pipeline is erroring with "SnapDataException: Timed out waiting for acknowledgment of consumed messages" error.
While investigating with pipeline statistics data, we have observed that the Pipeline execute snap is sending one lesser output document than the number of input documents. Example: We have 4 documents flowing from Kafka consumer. We are using Pipeline execute snap right after reading data from Kafka. Now the statistics say there are 4 input documents. But the output says 3 documents.

Contrary to this we can see 4 successful executions in the dashboard and the JCC log data confirm us with 4 triggers and 4 completion of child pipelines. This incident has been occurring intermittently. Has anyone experienced this issue? If so, please suggest us for a solution. 

Thank you!! 

11 Replies

    • pradhyumna_r's avatar
      pradhyumna_r
      New Contributor
      2 years ago

      The error is we can see is on the Kafka Consumer saying "Timed out waiting for acknowledgment of consumed messages"

    • ptaylor's avatar
      ptaylor
      Employee
      2 years ago

      Hello,

      Is the Pipe Exec configured to execute all children in the same local node? The Acknowledge snap must execute on the same node as the Consumer. So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.

      To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.

      • pradhyumna_r's avatar
        pradhyumna_r
        New Contributor
        2 years ago

        Hi ptaylor, Please find our response to the queries shared.
        Q: Is the Pipe Exec configured to execute all children in the same local node?
        A: Yes they are.

        Q: The Acknowledge snap must execute on the same node as the Consumer.So you either need to configure the Pipe Exec to Execute On LOCAL_NODE, or output the document from the child pipeline and move the Acknowledge snap to the parent.
        A: Yes they both are in snap parent pipeline. - Child pipeline is configured to run on LOCAL_NODE

        Q: To clarify one thing Kory said, when Auto Commit is disabled, the default Acknowledge Mode is "Wait after each record" but can be changed to "Wait after each batch of records". In that mode, the Acks can occur in any order. This also gives much better performance.
        A: Yes, Auto Commit is disabled, and we have also configured "Wait after each batch of records".

        Awaiting for your response, Thanks!

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    pradhyumna_r - The Kafka Consumer snap will only output one message at a time, which must be confirmed before the next message is sent.  I would check your child pipeline to ensure that it is returning all data back to the parent.  For example, is there a filter or an error in the child that prevents the message from returning back?

    • pradhyumna_r's avatar
      pradhyumna_r
      New Contributor
      2 years ago

      Hi koryknick Yes, we have configured "Wait after each batch of records". And please find our child pipeline configuration snip. As we are not filtering for any objects but are flagging the pipeline for Retry on Write failure to destination path. 

       

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    pradhyumna_r - please check the Dashboard (or Monitor) and look at the execution statistics on the child pipeline for one of your jobs that have failed.  My guess is that you have an unexpected error that is not returning the data back to the parent.  

    If not, then please provide your configuration of the Pipeline Execute snap.  One question here - are you intending that each message off of the Kafka queue is written to it's own zipfile?

    Or you could open a support ticket for assistance if you feel that the behavior is incorrect.

  • pradhyumna_r's avatar
    pradhyumna_r
    New Contributor
    2 years ago

    1. The Dashboard indicates all the child pipelines have completed successfully and doesnt have any errors. 

    Q : If not, then please provide your configuration of the Pipeline Execute snap. One question here - are you intending that each message off of the Kafka queue is written to it's own zipfile?
    A: Yes. We are reading and splitting. 
    Please find our Config for Pipeline Execute below,

     

     



    • pradhyumna_r's avatar
      pradhyumna_r
      New Contributor
      2 years ago

      Hi koryknick, please find our answers in the above reply and help us with your insights on it. 
      Thanks!

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    pradhyumna_r - I believe this is just an issue of the document being filtered in your child pipeline execution and the parent not able to acknowledge it.  Since I can't see the details in your provided screenshots, I recommend that you open a support ticket through your company's SnapLogic contacts on this issue for further assistance.