Kafka consumer Skip messages when fail

[neeraj_sharma]

Hi ,

We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails.
for example: in a run Kafka consumer is supposed to read offset 3,4,5 but it pipeline fails so it skips these offsets in the next run.

I tried using Kafka acknowledge snap after data inserted in the database it always time out.
anybody has any solution

[neeraj_sharma]

I think there is no direct way to “Inner join” both Input as the Snowflake Bulk Load snap does not passthrough what is inserted or there are no way to pass through the parent snap`s documents.
I think for now “Merge Join” is the closest to verifying the data inserted into the Snowflake or not, if I am afraid about Validating a pipeline would “Skip” some data so I should change the “Snap Execution” of Kafka Aknowladge snap to Execute only.

let me know if you think the same.

[ptaylor]

Yes, just change the Ack’s mode to Execute Only.