I have a very small pipeline (3 snaps) that I’m reading from a SQL table and writing the results to a new table in the same DB. I keep getting an error: Lost contact with Snaplex node while the pipeline was running. The select statement is pulling 250+M records and I’m using the Azure bulk insert to write. In order to avoid this error I keep having to reduce the batch size, from 20K to 10K to now 8K. Any thoughts on what could be causing the error?
Hi @msalter ,
I assume that root cause is the memory usage. In my case I was receiving below error message:
It was resolved by using Shard Offsets snap, and transferring the data in Threads, using Pipeline execute snap.
Thanks @ljupcho_machkovski . I thought that may be the case too but CPU never peaks above 60% and memory never peaks above 35%.