Hi Team,
I had a requirement where I have to use a sort snap and this sort needs to sort 50000 documents. But after processing 3057 documents, this snap kind of stuck keeps on validating but not processing the next documents.
It’s running for 24 hours. Sort order I m using as global. Could u pls let me know the reason why it’s not completing.
If I process less documents same pipeline works fine. What’s the option we xam use to fix this.
Regards
Hirak
What snaps do you have downstream of the sort? If you disconnect the output of the sort, does it complete?
Downstream we had a copy snap. We didn’t tried yet by delinking it. We can try that.
But what’s the reason why this is not completimg could u pls explain.
Thanks
Hirak
It’s difficult to determine why it isn’t completing from the description alone. As Kory pointed out, a downstream Snap may be a factor. Because the Sort Snap can cache records to disk based on the memory % at the bottom of the Sort Snap, that could be a factor as well. What is the memory % at the bottom of your Sort snap? How many fields are you trying to sort? Is the Snaplex undersized or under heavy load, either of which might leave the Sort snap with insufficient resources to process the data. How large are the 50,000 records?
If you can share any of that additional information here without exposing sensitive data we can try to assist here. If any of those questions lead you to sensitive data it would be better to contact our Support team for further investigation.
@hirakangshu.nag - please look at your pipeline execution statistics and find the light blue duration bar - this is where the pipeline is actually spending it’s time in execution. More than likely, the Sort snap is dark blue indicating that it is waiting on output to the downstream snaps. My guess is that it is “hung” at a database read/update or a Join snap using unsorted input.
It likely succeeds with smaller datasets because the end of input is hit before the downstream snap that is causing the deadlock starts buffering documents between snaps.
Hi there,
Was there a resolution for this?
I have an event driven pipeline but the documents are not being passed through the pipeline and are stuck at the Sort snap. Removing this affects are data downstream as it becomes unstructured. The Sort snap proceeds a Group by Field snap.
Seems like this is a SnapLogic flaw?
Hello Nai,
What do you mean by “an event driven pipeline”? For a Sort snap to complete and produce meaningful output, it must have a complete data set, so the input stream must end. My guess is that the input to the Sort isn’t ending.
