Solved
Kafka consumer Skip messages when fail
Hi ,
We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails.
for example: in a run Kafka consumer is supposed to ...
Forum Discussion
A merge join will merge available documents from both inputs but if one input provides fewer documents than the other (or no documents) it will just pass through the additional documents to the output without merging them with anything. Try changing the join type to Inner and select the left path and right path to join on a matching field in the input documents.
I think there is no direct way to “Inner join” both Input as the Snowflake Bulk Load snap does not passthrough what is inserted or there are no way to pass through the parent snap`s documents.
I think for now “Merge Join” is the closest to verifying the data inserted into the Snowflake or not, if I am afraid about Validating a pipeline would “Skip” some data so I should change the “Snap Execution” of Kafka Aknowladge snap to Execute only.
let me know if you think the same.
Yes, just change the Ack’s mode to Execute Only.
