Snowflake Bulk insert just stuck, do not process any data
I have seen that sometimes my Kafka to snowflake bulk insert is just stuck it does not process anything, neither it fails at the “Snowflake Bulk Insert snap”, like if there is a snowflake outage for 5 min, it gets stuck even after outgate is over, in that case, I manually restart the pipelines. is there a way to stop the pipeline if it just stuck at “Snowflake Bulk Insert snap” for X minutes? or any other better way to solve this.1.4KViews0likes0CommentsKafka consumer Skip messages when fail
Hi , We pull the data from Kafka and put it and the database, but we realized the Kafka consumer skips the data/offsets if the pipeline fails. for example: in a run Kafka consumer is supposed to read offset 3,4,5 but it pipeline fails so it skips these offsets in the next run. I tried using Kafka acknowledge snap after data inserted in the database it always time out. anybody has any solutionSolved19KViews0likes16Comments