Wassim
New Contributor
Joined 5 years ago
User Profile
User Widgets
Contributions
Re: Problem with copy and join
ptaylor: This is actually a known issue (SWAT-3096) that we’re working on a fix for. It happens when there are at least 1024 records being copied by the Copy snap, for reasons that are a bit difficult to explain. Until we have a fix, there are at least three workarounds: Swap the order of the inputs to the Lookup snap, so that the output of the Aggregate is the first input rather than the second. Insert a Sort snap right after each output of the Copy snap. It won’t work if you put the Sort before the Copy. In this workaround, the point of the Sort snaps isn’t to sort the data, which might already be sorted – it’s to essentially create independent buffers of the data from each of the Copy snap’s output views. Replace the Lookup with a Join, and set the Sorted streams property to Unsorted . thank you very much @ptaylor3.4KViews0likes0CommentsRe: Problem with copy and join
@Spiro_Taleski Thank you for the answer. i did try join also. i am aware of all that. i have 6000 rows. to aggregate to join to my first result and its not even moving its like this if i duplicate my snaps and make the aggregate and the join to the intial snaps it will take less than a minute. Regards3.4KViews0likes1CommentProblem with copy and join
Hello, I am working on this pipeline The problem is with this part I copy my result set. I keep my first output And aggregate my second output And finally I want to take those aggregate results using a lookup. This make my pipeline run endlessly. And if I remove the lookup (or join) and write to two different files it takes less than 2 minutes. I think may be its because the two outputs are the image of the same result set. Could you please tell me if you have seen this issue before and how to treat it. Thank youSolved9.5KViews0likes11Comments