Pipeline execution in series
Hi,
I have to execute two pipelines in series, currently using two pipeline execute snaps connected to each other, when I execute the pipeline these pipeline execute snap will execute both child pi...
Forum Discussion
Can I ask why they need to be run exclusively and cannot be run in parallel? Are they accessing some common resource?
There’s no explicit support for this in the PipelineExecute snap. The simplest option would be to put the two PipeExec snaps in their own pipeline and have the parent run that new pipeline. The root will wait for the new middle pipeline to finish before running the next one and the middle will wait for the two children to finish before it returns.
Yes, they are using the same resource. I am working on a load plan which has to execute 10 pipelines. There is a performance issue when executed all 10 parallel , broke this to 2 pipe line with 5 of each. now when I am executing these two pipelines using pipeline execute, they are getting into parallel again.
Can I ask what the issue was? Ideally, the snaps/platform should be able to cope and try to avoid failing.
I am working on loading data from CSV to Oracle DB, there is a performance issue when all pipelines are executed in parallel and also a scenario where the target table B is dependent on stage table A. So these leads for me to try in Series, like completion of TableA all docs and then Table B pipeline will start.
