Merge multiple documents to an array in ultra pipeline

[thilkip]

We need to merge multiple documents to single array and looking for help. Since it is ultra task, we are running out of options…

Input data:

[
{“Record”: “1”},
{“Record”: “2”},
{“Record”: “5”},
{“Record”: “4”},
{“Record”: “6”}
]

Expected Output:
[ “response”: [{“Record”: “1”}, {“Record”: “2”}, {“Record”: “5”}, {“Record”: “4”}, {“Record”: “6”}]
]

[koryknick]

I don’t have a DB2 instance to test with at the moment, but it appears you could do the same thing with different syntax:

ibm.com

JSON_ARRAY

The JSON_ARRAY function generates a JSON array either by explicitly listing the array elements or by using a query. If no JSON-expression is provided, the fullselect returns no values, or all values are null and ABSENT ON NULL is specified, an empty...

[koryknick]

It should be noted that I’m recommending that you push the creation of the array back to the source database because you are trying to prevent calling a child pipeline, which is the recommended pattern for aggregating data in an Ultra pipeline. This is required because Ultra is a streaming service of one record in, one record out. So any type of function that aggregates data (including Sort, Group By *, etc) will be stuck waiting on end of upstream input which will never come because of the nature of the data flow in Ultra. This is also why your Script snap isn’t working the way you were expecting - it’s waiting on the end of input, which never comes in an Ultra pipeline.

[bojanvelevski]

Hi again @thilkip,

This task of yours was very intriguing, so I dedicated quite sometime experimenting and I found a solution that works in development mode, with snaps supported by Ultra Task. Here is the pipeline:

Documents_to_Array_Format.slp (7.3 KB)

Here is the final result:

I am pretty curious about this one, can you please send feedback on the pipeline’s behaviour?
In addition I am sending you a Python script, that is also tested and works. I know that you allready tried with a script but I am offering some options. Thanks,

Doc_to_Array.txt (1.6 KB)

Best Regards,
Bojan

[thilkip]

bojanvelevski:

Hello Bojan

Thank you for your time and help. We did validate the script in ultra task and saw the same behavior like earlier.

We will test the other approach using Formatter snap and let you know.

Thank you again

[vamsi]

@thilkip Have you been able to achieve this scenario using the Script in the Ultra sscenario. We ran in to the same problem