How to create new array if there are more than 100 elements in the array

Designing Pipelines
,
1

Hi , currently we have an array that contains more then 100 elements , approximately 168. Is it possible with some transformation in the mapper to create another array that will contain the rest 68 elements ?

We would have 1 array with 100 elements and the other should contain the rest of the elments.

Or we can split the array if there are more then 100 documents , that would work too.

3

Hi @Igor_Bozhinovski ,

You can take a look at Group By N snap, I think it will help you with this issue.

You can set Group Size to 100 and you will have two groups as output.

BR,
Marjan

1 Like
4

Thanks Marjan , the array is part of the object and the Group By N snap doesnt seems to help.
The array looks like this:

SerialNumbers":[
{
“SerialNo”:“123”,
“SerialNo”:“456”,
}
]

Regards,
Igor

5

@Igor_Bozhinovski - I think @marjan.karafiloski has a great idea here. Use the JSON Splitter to first break the array into individual documents, then use the Group By N snap to regroup them into the size you want. This will handle the case when your array grows beyond 200 so there are no groups that will be larger than 100.

1 Like
6

Hi Igor,

Then maybe Group by fields snap is the one you are looking for?

You can choose by which object to group by.

Let me know if this helps you.

BR,
Marjan

7

Hi @Igor_Bozhinovski,

Good day, you can also use the slice() function of an array… see sample below where $object have 210 key-value pair and $array has a length of 210

For grouping key-value pair by the 100s

‘,’.repeat(Math.floor($object.keys().length / 100)).split(‘,’).map((x,i)=> $object.filter((v,k)=> $object.keys().slice((i * 100), (i + 1) * 100).indexOf(k) != -1))

For grouping array by 100s
‘,’.repeat(Math.floor($array.length / 100)).split(‘,’).map((x,i)=> $array.slice((i * 100), (i + 1) * 100))

image
image1025×612 73.4 KB

~Alchemiz

8

Thanks Alchemiz , the array is now spliited in two. One array with 100 elements and the other with the remaining elements.

This is the current output :

"ShipmentCreateDate": "01/01/2023",
"ShipmentID": "1",
"SerialNumbers": [
  [
    {
      "SerialNo": "123"
    },
	{ "SerialNo":"456"
    }
  ],
  [
	{
	  "SerialNo": "101"
	},
	{
	  "SerialNo": "102"
	}
  ]
 ]

I was wondering if something like this output is possible :

"ShipmentCreateDate": "01/01/2023",
"ShipmentID": "1",
"SerialNumbers": [
  [
    {
      "SerialNo": "123"
    },
	{ 
	  "SerialNo":"456"
    }
  ]
"ShipmentCreateDate": "01/01/2023",
"ShipmentID": "1",
"SerialNumbers": [
  [
	{
	  "SerialNo": "101"
	},
	{
	  "SerialNo": "102"
	}
  ]
 ]

Where the first output would be the array with the 100 elements and the remaining fields that are not part of the array and the second output would be the array with the remaining elements and the fields that are not part of the array.

Regards,
Igor

9

Hi Sir,

See attached PoC pipeline. Hope this helps

image
image866×314 25.7 KB

canvas_2023_01_10.slp (11.7 KB)

Thanks,
EmEm

10

Thanks this is what i was exactly looking for.