Solved
Kafka consumer Skip messages when fail
Hi ,
We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails.
for example: in a run Kafka consumer is supposed to read offset 3,4,5 but it pipeline fails so it skips these offsets in the next run.
I tried using Kafka acknowledge snap after data inserted in the database it always time out.
anybody has any solution
Try using this:
Wildcard is your file format, it will go into a single route for one format.
