09-07-2021 09:34 AM
Hello,
I am working on this pipeline
The problem is with this part
I copy my result set.
I keep my first output
And aggregate my second output
And finally I want to take those aggregate results using a lookup.
This make my pipeline run endlessly. And if I remove the lookup (or join) and write to two different files it takes less than 2 minutes.
I think may be its because the two outputs are the image of the same result set.
Could you please tell me if you have seen this issue before and how to treat it.
Thank you
Solved! Go to Solution.
09-08-2021 03:15 PM
Hi Wassim,
This is actually a known issue (SWAT-3096) that we’re working on a fix for. It happens when there are at least 1024 records being copied by the Copy snap, for reasons that are a bit difficult to explain.
Until we have a fix, there are at least three workarounds:
Sorted streams
property to Unsorted
.09-07-2021 11:47 AM
Hi @Wassim,
Try first to sort the data before you send it to the join.
How many records did you get before coping the records ?
If it’s only one, then instead of using Join snap try it with Gate snap. You will get same result as it is with Join. If there are more records and you are joining by some conditions then you can not use Gate.
Regards.
09-07-2021 01:18 PM
Hi Viktor,
Thank you for the answer.
I have 6k rows so i cant use the gate.
I sorted on the same field i group with and it doesnt help.
09-07-2021 04:26 PM
Did you mean Join Snap when you referred to Lookup snap here ? Also what is the output from the Aggregate Snap and Mapper after that ?
09-08-2021 07:27 AM
Hi skatpally thank you very much for the answer. its a lookup snap
Here is the aggregate
and here is the mapper after the aggregate
thank you