Checking for existence of an array in a mapper
Hello. Im looking for a way of dealing with JSON documents that are sometimes recognised as an array, and sometime not. In the below example, i need to manipulate the “message” element to remove certain messages before passing these on to another web hook. We have achieved this using routers to check for the existence of an array, and they join back. However we need this as an ultra pipeline, so cannot use a join.
the current solution looks like this…
essentially what i am going here is copying the document, routing, array or non array, manipulating and joining back. I need to be able to do all this in a single path in a series of mappers, without using a join. Any help would be greatly appreciated.
SL12345 - You can also use the Pipeline Execute snap and set the Batch Size value to your desired number of records in the target. Create a child pipeline that is just a CSV Formatter and File Writer snap and call that child in your Pipeline Execute.
Basically, this passes the number of records specified in the Batch Size property to an instance of the child pipeline which can create a file of those records and finish, then the parent will start a new child with the next "batch" of records, and will keep iterating until all input records are consumed. Simple data chunking of your original file.
Hope this helps!
