How to render distinct elements only in an array with duplicates

[mohamadelmardin]

I have a JSON document that has inside an array with duplicate values such as [1,1,3,3,6,1,2,1,2,100]. I am trying to transform/map it from source into target by returning only the distinct values such as [1,3,6,100]. How can I achieve that using a Mapper snap or if any other snap. For example when using jsonPath() function to specify a part of JSON document that has an array, is there a function similar to .unique() that could be attached to return only unique array elements?
The Unique snap works only on entire documents/rows not array elements within a document.

[cjhoward18]

Hi @Anjali

item is the current item it is iterating over in the array, pos is the position in the array for this item and a is the array itself.

The expression to remove duplicates is using the indexOf() to collect a single instance of each unique element in the filter() method since indexOf() returns the first found index for each unique value each time.

[srao]

Hi, I have an complex JSON array and I have to remove the duplicates based on a field. For Ex:
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:a,…} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ],
How can I remove the duplicate array elements based on the ProductNum Field to produce the below JSON
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ]

I have tried the above logic and it didnt work, as other fields in this json structure.

[tstack]

Same idea, except you’d want to use the ‘findIndex()’ method with a custom predicate instead of the ‘indexOf()’ method, like so:

$arr.filter((item, pos, a) => a.findIndex(elem => item.ProductNum == elem.ProductNum) == pos)

So, the ‘findIndex()’ method will walk through the array and evaluate the predicate function (elem => item.ProductNum == elem.ProductNum) on every element in the array. If a match is found, the index is returned and, if it’s the same index as the current element being examined in the filter, then we add the element to output list.

[Tanmay_Sarkar]

Hello,

Is there a way we can totally remove the duplicates values from the array?

For instance if my array looks like [1,2,3,1,3,4,5], I want them separated out with each other like [2,4,5] and [1,1,3,3]

[Anjali]

Can you please explain how this filter function is working? what is item , pos and a here and how there are functioning?