Solved
Kafka consumer Skip messages when fail
Hi ,
We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails.
for example: in a run Kafka consumer is supposed to ...
Forum Discussion
Hi @ptaylor,
Appreciate you in depth response.
I agree I should use “Snowflake bulk Load” snap with Kafka Consumer`s Batch Mode option, I usually Validate pipeline and then I select the $variable this ensures correctness, but the pipeline did not work last time, I re-validated and it start to working.
So then I replace snowflake insert to Snowflake bulk insert, but then again it started to timeout
So, I thought Bulk insert might work with “Kafka Consumer`s” Batch acknowledge property but it does not work, it gives Time Out.
I think I am missing something to add, can you please see the properties what I am missing here and please add pipeline as well if you are trying it out.
Attaching all 3 pipeline, I added Extra mappers etc for debug purpose only.
Bulk insert with Batch not Working Pipeline_2022_02_22.slp (17.3 KB)
Bulk insert not Working Pipeline_2022_02_22.slp (12.1 KB)
Working Pipiline_2022_02_22.slp (11.4 KB)
