Merge documents into one big document


Currently when I use Oracle select/Oracle execute snaps to query data from Snaplogic, it will return multiple documents as output from those snaps(one row per doc).

Is there a way i can merge them together into one json array or one doc?


Did you look into Group By N with a setting of 0!! Try it and let us know.

No, groupby N has limitation when using Ultra task.
So for us we are trying to build high performance API with Ultra task. GroupbyN can achieve the goal but as it can only work in Triggered/Schedule tasks.

Try moving that group by N to a child pipeline and see how it works for Ultra.

Most of the snaps (group by, group By N, aggregate snaps) work in child pipelines in Ultra mode as long as you return one document to the top level parent pipeline.

Thats a workaround we have tried, but API performance is not as good as what we expected.
From Snaplogic professional, it seems doing parent-child pipeline will add some overhead even with Ultra task.

One way of achieving faster response is to provide limit and offset to the callers using this Ultra API.

LIMIT and OFFSET can be easily implemented in database queries!

You can easily implement this and thereby dont expose the whole set of data when not needed.

No, in our case, we have to expose all data from one select * query.

Or you could try doing the group by in python since “Script” snap does support ultra. I know this isn’t ideal but at least this might satisfy your business requirements.

Yes, Script is the only way we can do this which can provide faster performance as well.
But we would avoid to use script or any snap development. That defeat the purpose of ‘Citizen integration’ as what Snaplogic advertised.

What is your expectation in terms of time and results?

200 docs in 1 sec or 3 secs or subseconds?

What config do you have? How many nodes/FMs?


So we got a script written by your professional. the whole pipeline with this script snap and ultra task only takes 200ms.

Remember when I say we used ultra task with groupbyN snap and Pipeline execute, it takes 3 to 4s.

Something definitely need a enhancement request. Performance is bad.

Yes, we have asked for a request

I think you meant by non-ultra task with groupbyN take 3-4 seconds which I think is what we can expect. There is a reason why there is ultra which is exactly design to offer sub-seconds respond time. I believe the more sensible enhancement request is to enable more functions like GroupbyN to be supported by Utra.


1 Like