Pipeline execution in series

Hi,

I have to execute two pipelines in series, currently using two pipeline execute snaps connected to each other, when I execute the pipeline these pipeline execute snap will execute both child pipeline parallely.

Please could you suggest the option of using pipeline execute snap executing in series.

Regards,
Hemanth

Can I ask why they need to be run exclusively and cannot be run in parallel? Are they accessing some common resource?

There’s no explicit support for this in the PipelineExecute snap. The simplest option would be to put the two PipeExec snaps in their own pipeline and have the parent run that new pipeline. The root will wait for the new middle pipeline to finish before running the next one and the middle will wait for the two children to finish before it returns.

Yes, they are using the same resource. I am working on a load plan which has to execute 10 pipelines. There is a performance issue when executed all 10 parallel , broke this to 2 pipe line with 5 of each. now when I am executing these two pipelines using pipeline execute, they are getting into parallel again.

If you mean you want all docs to go through the first before running the second, you could do a tail snap in between the pipeline executes to grab the last document out of the first before running the second. If you need the document output, so basically wanting to do batch as opposed to SnapLogic’s typical way of streaming data through, you could instead put a sort snap with unsorted input which will force the snap to read all documents first before sorting. Note that means all the documents will remain on the server for a time while they are gathered to sort, so would need to be sure you have the server resources to do so.

Can I ask what the issue was? Ideally, the snaps/platform should be able to cope and try to avoid failing.

Thank you, The Tail snap worked for me.
I am working on loading data from CSV to Oracle DB, there is a performance issue when all pipelines are executed in parallel and also a scenario where the target table B is dependent on stage table A. So these leads for me to try in Series, like completion of TableA all docs and then Table B pipeline will start.

I am working on loading data from CSV to Oracle DB, there is a performance issue when all pipelines are executed in parallel and also a scenario where the target table B is dependent on stage table A. So these leads for me to try in Series, like completion of TableA all docs and then Table B pipeline will start.