Solved
Kafka consumer Skip messages when fail
Hi ,
We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails.
for example: in a run Kafka consumer is supposed to ...
Forum Discussion
Yes, Input0 is having the “$message” output from “Snowflake Bulk Load” Snap.
“Snowflake Bulk Load” is Execute Only -
So, if the insert has not been executed so it should not pass the message from input0 and join should not perform.
The “Join” snap`s properties look like-
Am I missing something here, for reference I am attaching both parent and child pipelines:
Parent-test_2022_03_17.slp (4.3 KB)
test_2022_03_17.slp (14.0 KB)
A merge join will merge available documents from both inputs but if one input provides fewer documents than the other (or no documents) it will just pass through the additional documents to the output without merging them with anything. Try changing the join type to Inner and select the left path and right path to join on a matching field in the input documents.
I think there is no direct way to “Inner join” both Input as the Snowflake Bulk Load snap does not passthrough what is inserted or there are no way to pass through the parent snap`s documents.
I think for now “Merge Join” is the closest to verifying the data inserted into the Snowflake or not, if I am afraid about Validating a pipeline would “Skip” some data so I should change the “Snap Execution” of Kafka Aknowladge snap to Execute only.let me know if you think the same.
Yes, just change the Ack’s mode to Execute Only.
