Forum Discussion

Tanmay_Sarkar's avatar
Tanmay_Sarkar
New Contributor III
5 years ago
Solved

Remove duplicate values from the JSON array

Hello All,

Hi, I have a JSON array and I have to remove the duplicates based on a field and then I want the non duplicate elements in one array and the duplicate values in an another array.

Input JSON:
[
{“pName”: “abc”,“iNumber”: 123 },
{“pName”: “def”,“iNumber”: 123 },
{“pName”: “xyz”,“iNumber”: 890 },
{“pName”: “jkl”,“iNumber”: 456 }
]

Required Output:
[
{“pName”: “abc”,“iNumber”: 123 },
{“pName”: “def”,“iNumber”: 123 }
]
[
{“pName”: “xyz”,“iNumber”: 890 },
{“pName”: “jkl”,“iNumber”: 456 }
]
Two separate JSON arrays one with non duplicate iNumber and another with the one which were duplicates.

I tried filter((item, pos, a) => a.findIndex(elem => item.iNumber == elem.iNumber) == pos), it didn’t give out the required result it still keeps one of the duplicate values.

  • Try using the filter method

    e.g.

    {}.merge({“Unique”: $array.filter((a,b,c)=> c.filter((x,y,z)=> a[‘iNumber’] == x[‘iNumber’]).length == 1)}, {“Dups”: $array.filter((a,b,c)=> c.filter((x,y,z)=> a[‘iNumber’] == x[‘iNumber’]).length != 1)})

4 Replies

  • bojanvelevski's avatar
    bojanvelevski
    Valued Contributor

    Hi there @Senthilnaga91.

    The Snaplogic Snaps will be suitable for this use case. You have all CRUD operations at your disposal. Let me know if you still need help on this.

    Regards,
    Bojan

    • Senthilnaga91's avatar
      Senthilnaga91
      New Contributor III

      I have provided more details in my previous reply. Please refer.

      • Abhishek_Soni37's avatar
        Abhishek_Soni37
        Contributor

        @Senthilnaga91 ,

        As @bojanvelevski mentioned you can use the Metadata snap pack to achieve it,

        Solution Approach:

        1. Create all tasks that you want to run dynamically as Scheduled.
        2. Read data from the source table with info on when to run a pipeline.
        3. Read the specific Task with the help of the Metadata snap pack.
        4. Assign new values and update the Task with the Metadata snap pack.

        Set values below in the pipeline parameter:

        source_proj: orgName/ProjectPath
        target_proj: orgName/ProjectPath
        taskToUpdate: Name of the task

        Sample pipeline:
        UpdateTaskInfo.slp (3.7 KB)

        Let us know if this worked.

        Cheers 🙂

  • PratapKr_Gayen's avatar
    PratapKr_Gayen
    New Contributor II

    I think your requirement is to trigger the a pipeline when ever there is a change in record in source table.
    For that you have to create pipeline to save the current count in SLDB and compare the count of record in current execution. This pipeline to compare and check the record count mismatch need to be schedule 15 or 30 min and if there is count mismatch, call the pipeline to load data to Sales table.

    • Senthilnaga91's avatar
      Senthilnaga91
      New Contributor III

      No, let me be more precise. Our client has given a schedule to run what pipeline on which day and time (nnot a fixed schedule). It’s maintained in a RDBMS table. So, to trigger a pipeline, the process has to look into the scheduling table and trigger the pipleline on a particular day and time accordingly.

      Example: There is a record in scheduling table,
      PipeName,Date,Time
      LoadSalesTable,06-27-2023,05:00 PM CST
      LoadEmployeeTable,06-27-2023,04:00 PM CST

      So, my question is how to automate the process to trigger the pipeline ‘LoadSalesTable’ on 06-27-2023,05:00 PM CST ?

      (PS: The schedule for ‘LoadSalesTable’ might change in the future and the process should accommodate without manual intervention).