Forum Discussion

Sowmya_Rayavara's avatar
Sowmya_Rayavara
New Contributor III
7 years ago

Sort snaps takes more time

Hi All,

I have four sort snaps which run parallel. Few times they are getting sorted very quickly(in seconds), especially first time when I ran the interface, but next time or second time onwards, it is taking more time almost few hours. On the next day when I ran the interface again it is completed in few seconds. Could anyone please help me to understand what is happening? Also there is no change in the interface and also there are no other interfaces running in the servers.

Regards,
Sowmya.

6 Replies

    • dmiller's avatar
      dmiller
      Former Employee
      7 years ago

      Off the top of my head, I don’t have an explanation. I defer to our developers.

  • shane's avatar
    shane
    New Contributor II
    6 years ago

    This is also effecting me.
    Sometimes very quick but after I run a few pipelines everything comes to a crawl.

    just under 5K documents shouldn’t take that long to sort.
    Only the 2 columns referenced in the sort are in the document as well.

    • tstack's avatar
      tstack
      Former Employee
      6 years ago

      What is the snap downstream from the Sort? The “Output time” statistic there shows that the Sort snap seems to be waiting on the downstream snap to do its work. The time taken to do the Sort was less than a second.

      • shane's avatar
        shane
        New Contributor II
        6 years ago

        The following snap is just a Unique snap which results (in that specific instance) with the same result as was fed in.

        After that is a simple JDBC Select snap that when I check query logs, the select queries took ~2.5seconds each, and were pending for about 3 mins prior to execution but that snap seems to have taken 24 mins to execute. I sat there waiting for these snaps to finish for about 20 mins after the queries completed according to my redshift query logs.
        Either way it seems to just hang there and I can’t determine what is causing the hang.

        This does seem to be getting into something not quite related to the topic though lol. I’ll have to dig in deeper but unfortunately I can’t see every query in the Redshift query logs (could be part of my issue). This time upon execution of the pipeline the Redshift query logs showed no select statements at all.