How to render distinct elements only in an array with duplicates

I have a JSON document that has inside an array with duplicate values such as [1,1,3,3,6,1,2,1,2,100]. I am trying to transform/map it from source into target by returning only the distinct values such as [1,3,6,100]. How can I achieve that using a Mapper snap or if any other snap. For example when using jsonPath() function to specify a part of JSON document that has an array, is there a function similar to .unique() that could be attached to return only unique array elements?
The Unique snap works only on entire documents/rows not array elements within a document.

The expression language is a subset of JavaScript, so you can usually start by looking for JavaScript-based solutions.

In this case, take a look at this post:

The first answer has the simplest solution. The only change you have to make is to convert the callback function to use the arrow-function syntax, like so:

$arr.filter((item, pos, a) => a.indexOf(item) == pos)

4 Likes

Thank you. That was helpful :+1:

Thanks @tstack! For anyone wondering what this looks like in the Designer::

Pipeline .slp file: 20170404_mapper-distinct-elements_2017_04_04.slp (3.0 KB)

1 Like

Hi, I have an complex JSON array and I have to remove the duplicates based on a field. For Ex:
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:a,…} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ],
How can I remove the duplicate array elements based on the ProductNum Field to produce the below JSON
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ]

I have tried the above logic and it didnt work, as other fields in this json structure.

Same idea, except you’d want to use the ‘findIndex()’ method with a custom predicate instead of the ‘indexOf()’ method, like so:

$arr.filter((item, pos, a) => a.findIndex(elem => item.ProductNum == elem.ProductNum) == pos)

So, the ‘findIndex()’ method will walk through the array and evaluate the predicate function (elem => item.ProductNum == elem.ProductNum) on every element in the array. If a match is found, the index is returned and, if it’s the same index as the current element being examined in the filter, then we add the element to output list.

1 Like