We need to merge multiple documents to single array and looking for help. Since it is ultra task, we are running out of options…
Input data:
[
{“Record”: “1”},
{“Record”: “2”},
{“Record”: “5”},
{“Record”: “4”},
{“Record”: “6”}
]
Expected Output:
[ “response”: [{“Record”: “1”}, {“Record”: “2”}, {“Record”: “5”}, {“Record”: “4”}, {“Record”: “6”}]
]
The documentation on the Fixed Width Formatter being Ultra compatible is incorrect and we have filed a defect on it. I just tested an Ultra task with the Fixed Width Formatter and received an “incompatible” error.
Suggested approach is still to:
One thing to keep in mind with Ultra - the latency to start a child pipeline is extremely low since the pipeline metadata is cached locally so there is no communication to the control plane to start the process. It is pretty close to having the snaps contained within the main pipeline. Have you tried this approach and tested timing?
I did a quick sample that consumes the request and starts a child that generates several documents and uses a Gate to accumulate them to an array and passes the document back to the parent for return to the client. Total execution is under half a second based on “curl” stats.
I compared this with a pipeline that simply generates an array using a JSON Generator (no child) and total execution time is over half a second… actually longer than with the child pipeline!
I’ll continue to monitor this post. Please keep me updated on your solution!
Hi @koryknick ,
I have a question considering your solution with the gate snap.
My situation:
I have a parent api pipeline picture below:
And it calls the child pipeline, picture below:
When I test it out as triggered task it works. But when I enable it as ultra task it doesn’t work. I already checked the snap that don’t support ultra tasks (gate, group by and json formatter if ‘format each document is not selected’). But somehow it takes a few minutes and always give the error:
can't parse JSON. Raw result:
Pipeline did not produce a valid response
Is my response wrong format? because in triggered task in looks good
Do you have a simple parent with child pipeline with the gate snap that works as an ultra pipeline ?
Because I cannot find my error what I do wrong
Regards
@JensDeveloper - I believe for Ultra you will want to remove the JSON Parser from the beginning of your parent pipeline. With Ultra using a FeedMaster, the document passed into the pipeline is already a JSON document, not a binary stream as with Triggered Task pipeline executions.
Hi,
Thanks for the answer @koryknick. but after that change it still takes a few minutes to wait on the response and it then gives just the orginal object because it errors. Does it have something to do that i cannot split the path_info when it is enabled as an ultra task?
When it’s in triggered task i get these results:
When I enable it then as ultra i get this:
Is it because it’s listening whole the time to a call that it cannot get the url?
Or has it something to do with my configuration of my pipeline execute?
