Hi ,
I have designed a pipeline where I need to apply duplication on specific fields instead of all to drive other fields . We have RowVal in oracle which does this job by including RowVal=1 . Can we implement the same in snaplogic . I give a sample file extract and the requirement .
Where the count is assigned based on Call_ID , Product_ID, Is_Deleted
combination . In the screenshot we have the value of 2 in the count because the entry is duplicated . If we have the count value like this we can easily take the de-duplicate values by filtering $Count==1 . Please guide/propose a method which solves this
Thanks
I was able to achieve the desired result like this:
I have sample data like this:
Then I use Group by Field snap
Then a Mapper with the following expression jsonPath($, "object[*]").map((val, index, arr) => val.extend({"Count": index == 0 ? 1 : 2})). This way you are adding a Count field to every object, if index is 0 meaning first object then add a count of 1, if index > 0 then Count will always be 2. Output:
Then you can use filter snap to filter out those records with count = 2.
Make sure you add a sort by snap before group by snap.
Edit:
Also, if you don’t need the Count field you can make it more simple with a mapper with the following expression jsonPath($, "object[*]")[0], the output will be the same as above except there is no Count field. This way you only get one object from the grouped records (meaning all other duplicates are ignored).
