Merge multiple documents to an array in ultra pipeline

[thilkip]

We need to merge multiple documents to single array and looking for help. Since it is ultra task, we are running out of options…

Input data:

[
{“Record”: “1”},
{“Record”: “2”},
{“Record”: “5”},
{“Record”: “4”},
{“Record”: “6”}
]

Expected Output:
[ “response”: [{“Record”: “1”}, {“Record”: “2”}, {“Record”: “5”}, {“Record”: “4”}, {“Record”: “6”}]
]

[koryknick]

The documentation on the Fixed Width Formatter being Ultra compatible is incorrect and we have filed a defect on it. I just tested an Ultra task with the Fixed Width Formatter and received an “incompatible” error.

Suggested approach is still to:

1. Use a child pipeline to query the database and aggregate
   or
2. Have the database return the JSON Array

One thing to keep in mind with Ultra - the latency to start a child pipeline is extremely low since the pipeline metadata is cached locally so there is no communication to the control plane to start the process. It is pretty close to having the snaps contained within the main pipeline. Have you tried this approach and tested timing?

I did a quick sample that consumes the request and starts a child that generates several documents and uses a Gate to accumulate them to an array and passes the document back to the parent for return to the client. Total execution is under half a second based on “curl” stats.

I compared this with a pipeline that simply generates an array using a JSON Generator (no child) and total execution time is over half a second… actually longer than with the child pipeline!

I’ll continue to monitor this post. Please keep me updated on your solution!

• Kory K

[JensDeveloper]

Hi @koryknick ,

I have a question considering your solution with the gate snap.
My situation:

I have a parent api pipeline picture below:

And it calls the child pipeline, picture below:

When I test it out as triggered task it works. But when I enable it as ultra task it doesn’t work. I already checked the snap that don’t support ultra tasks (gate, group by and json formatter if ‘format each document is not selected’). But somehow it takes a few minutes and always give the error:

can't parse JSON.  Raw result:

Pipeline did not produce a valid response

Is my response wrong format? because in triggered task in looks good

Do you have a simple parent with child pipeline with the gate snap that works as an ultra pipeline ?
Because I cannot find my error what I do wrong

Regards

[koryknick]

@JensDeveloper - I believe for Ultra you will want to remove the JSON Parser from the beginning of your parent pipeline. With Ultra using a FeedMaster, the document passed into the pipeline is already a JSON document, not a binary stream as with Triggered Task pipeline executions.

[JensDeveloper]

Hi,

Thanks for the answer @koryknick. but after that change it still takes a few minutes to wait on the response and it then gives just the orginal object because it errors. Does it have something to do that i cannot split the path_info when it is enabled as an ultra task?

When it’s in triggered task i get these results:

When I enable it then as ultra i get this:

Is it because it’s listening whole the time to a call that it cannot get the url?

Or has it something to do with my configuration of my pipeline execute?

[koryknick]

@JensDeveloper - Ultra tasks consume incoming documents. I believe what you want to do is POST data to the Ultra task, not pass it as URL query parameters. See the Ultra Pipeline best practices documentation final bullet point in regards to your situation with trying to parse the pipeline parameters:

• You cannot change Pipeline parameters at run time. These parameters are specified in the Pipeline itself or when the task is created. They are set at pipeline startup, which only happens on initiation, not per document.