Forum Discussion

New Contributor

5 years ago

Solved

Problem with copy and join

Hello, I am working on this pipeline The problem is with this part I copy my result set. I keep my first output And aggregate my second output And finally I want to take those a...

sshaik
3 years ago
Solved this scenario in a different way.
Insert handled by using inner join.
For Update
Here I have to update 2 columns($status, $ID) based on 6 key columns matching data.
So, I have read the target table as reference table($col_Ref) and regular/new data as left side data and performing an inner join based on 6 key columns.
There I found the matching data.
I put 1 filter to sort $status=“Active”
I just mapped 2 ($status hardcoded as “INACTIVE” I have to update the $status column if $stat=“Active”, $ID taken from filter ) columns which I want to update in the Target Table.

Finally in Oracle Update snap mentioned Schema and table name.
In where clause $Status=“Active”.

This was one of the ways that I found for resolving this issue.

If we have any other ways to handle this, then it could be helpful.

ptaylor

Employee

5 years ago

Hi Wassim,

This is actually a known issue (SWAT-3096) that we’re working on a fix for. It happens when there are at least 1024 records being copied by the Copy snap, for reasons that are a bit difficult to explain.

Until we have a fix, there are at least three workarounds:

Swap the order of the inputs to the Lookup snap, so that the output of the Aggregate is the first input rather than the second.
Insert a Sort snap right after each output of the Copy snap. It won’t work if you put the Sort before the Copy. In this workaround, the point of the Sort snaps isn’t to sort the data, which might already be sorted – it’s to essentially create independent buffers of the data from each of the Copy snap’s output views.
Replace the Lookup with a Join, and set the Sorted streams property to Unsorted.

Wassim
New Contributor
5 years ago
Copy Link
Report
ptaylor:

This is actually a known issue (SWAT-3096) that we’re working on a fix for. It happens when there are at least 1024 records being copied by the Copy snap, for reasons that are a bit difficult to explain.

Until we have a fix, there are at least three workarounds:

Swap the order of the inputs to the Lookup snap, so that the output of the Aggregate is the first input rather than the second.

Insert a Sort snap right after each output of the Copy snap. It won’t work if you put the Sort before the Copy. In this workaround, the point of the Sort snaps isn’t to sort the data, which might already be sorted – it’s to essentially create independent buffers of the data from each of the Copy snap’s output views.

Replace the Lookup with a Join, and set the Sorted streams property to Unsorted .

thank you very much @ptaylor
vgautam64
New Contributor III
3 years ago
Copy Link
Report
I too faced this exact problem recently and this thread proved really helpful. Thanks a lot!

Considering this post is more than 1.5 years old now, what is the status on the fix for this issue?