Hello All,
I got stuck with some issue, if you can help.
I am trying to compare two streams of documents using Diff snap for that I am using Sort Snap to sort the stream of Documents(43808 docs) and it got stuck at 3075th Document and is not progressing further, moreover there is no error message. I have attached the screenshots of the pipeline for reference along with Dashboard execution statistics.
Any inputs is appreciated.
Thank You
Looking at just what has been shared here, three of the Snaps in your Pipeline Statistics show long purple bars indicating that a downstream system has effectively become a bottleneck in processing the data - Sort_ControlTablePrimaryKey, Copy_ControlTablePrimaryKey, and Mapper_ControlTablePrimaryKey. Could you share more of the Pipeline Statistics so we can see how the PostgreSQL database is doing?
If the PostgreSQL instance is the bottleneck (and again, that’s just my thought based on what is here, it may not be the cause at all), you could try increasing the Batch size in the PosgreSQL account. The default value is 50 so that might be one lever you can pull for performance.
Here is a screenshot with the Batch and Fetch sizes increased from their values of 50 and 100 respectively:
