Sort snaps takes more time

Hi All,

I have four sort snaps which run parallel. Few times they are getting sorted very quickly(in seconds), especially first time when I ran the interface, but next time or second time onwards, it is taking more time almost few hours. On the next day when I ran the interface again it is completed in few seconds. Could anyone please help me to understand what is happening? Also there is no change in the interface and also there are no other interfaces running in the servers.


1 Like

@tstack, @del , @dmiller Can you help me in answering this? Thanks in Advance

Off the top of my head, I don’t have an explanation. I defer to our developers.

This is also effecting me.
Sometimes very quick but after I run a few pipelines everything comes to a crawl.

just under 5K documents shouldn’t take that long to sort.
Only the 2 columns referenced in the sort are in the document as well.

What is the snap downstream from the Sort? The “Output time” statistic there shows that the Sort snap seems to be waiting on the downstream snap to do its work. The time taken to do the Sort was less than a second.

The following snap is just a Unique snap which results (in that specific instance) with the same result as was fed in.

After that is a simple JDBC Select snap that when I check query logs, the select queries took ~2.5seconds each, and were pending for about 3 mins prior to execution but that snap seems to have taken 24 mins to execute. I sat there waiting for these snaps to finish for about 20 mins after the queries completed according to my redshift query logs.

Either way it seems to just hang there and I can’t determine what is causing the hang.

This does seem to be getting into something not quite related to the topic though lol. I’ll have to dig in deeper but unfortunately I can’t see every query in the Redshift query logs (could be part of my issue). This time upon execution of the pipeline the Redshift query logs showed no select statements at all.

What about the snaps that follow that, are there any with a long blue bar in the duration column that would indicate a snap that is taking a long time to execute? Maybe include a screenshot of the execution statistics. The Execution Statistics documentation might help you understand the details.