How to render distinct elements only in an array with duplicates

Question

I have a JSON document  that has inside an array with duplicate values such as [1,1,3,3,6,1,2,1,2,100]. I am trying to transform/map it from source into target by returning only the distinct values such as [1,3,6,100]. How can I achieve that using a Mapper snap or if any other snap. For example when using jsonPath() function to specify a part of JSON document that has an array, is there a function similar to .unique() that could be attached to return only unique array elements?
The Unique snap works only on entire documents/rows not array elements within a document.

tstack · Answer

The expression language is a subset of JavaScript, so you can usually start by looking for JavaScript-based solutions.
In this case, take a look at this post:

stackoverflow.com

Remove duplicate values from JS array

javascript, jquery, arrays, duplicates

asked by

kramden88
  
  on 02:53PM - 10 Feb 12 UTC

The first answer has the simplest solution.  The only change you have to make is to convert the callback function to use the arrow-function syntax, like so:

$arr.filter((item, pos, a) =&gt; a.indexOf(item) == pos)

robin · Answer

Thanks @tstack! For anyone wondering what this looks like in the Designer::

Pipeline .slp file: 20170404_mapper-distinct-elements_2017_04_04.slp (3.0 KB)

tstack · Answer

Same idea, except you’d want to use the ‘findIndex()’ method with a custom predicate instead of the ‘indexOf()’ method, like so:
$arr.filter((item, pos, a) =&gt; a.findIndex(elem =&gt; item.ProductNum == elem.ProductNum) == pos)

So, the ‘findIndex()’ method will walk through the array and evaluate the predicate function (elem =&gt; item.ProductNum == elem.ProductNum) on every element in the array.  If a match is found, the index is returned and, if it’s the same index as the current element being examined in the filter, then we add the element to output list.

mohamadelmardin · Answer

Thank you. That was helpful  :+1:

srao · Answer

Hi, I have an complex JSON array and I have to remove the duplicates based on a field. For Ex:
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:a,…} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ],
How can I remove the duplicate array elements based on the ProductNum Field to produce the below JSON
[ {PartnerID:1127839, ProductNum:a, …} ,
{PartnerID:1127839, ProductNum:b,…} ,
{PartnerID:1127839, ProductNum:c,…} ]
I have tried the above logic and it didnt work, as other fields in this json structure.

Forum Discussion

How to render distinct elements only in an array with duplicates

Recent Discussions

Ingesting Data into Veeva Vault CRM via SnapLogic – Alternatives to SFDC Snaps

SFTP: Worked ok then settings changed not not working

SQS Consumer snap is ready all message at pipeline startup

Salesforce Update REST API Throughput

Using Windows Authentication with SQL Server Accounts