How to wait child pipeline till parent pipelines are completed

[Harriesh]

How to wait child pipeline till parent pipelines are completed. Currently the child pipelines are getting executed before the parent execution done with all the documents.
Because of that it is having memory issue.

Is there any wait to wait the child pipeline till all the documents execution and then call the child pipeline.

[stephenknilans]

First of all, don’t have preceding tasks send any output to following tasks, unless it is the end of their process. If you do, the following tasks(which would deal with the child records), will run. Of course, you COULD provide aggregates of keys and accept that data to join in the child. This allows feedback, aggregation, RI validation, and processes children only after the parents are processed. Of course there is no value to that other than to isolate processes, and allow distributing among multiple nodes. OH, and doing that ALSO limits memory consumption if you use multiple JVMs, or multiple nodes.

[Harriesh]

stephenknilans:

on if you use multiple JVMs, or multiple nodes

Let me try out that option…

Thanks.

[aleung]

Harriesh:

child pipeline till all the documents execution and then call the child pipeline.

You could also try writing the output to a file / db from the parent pipeline, that way you can force all processing is done prior to initiate the child.

[stephenknilans]

I probably should have said that. I have one case where I am doing precisely that. But I am kicking off the tasks to handle parameters and order, and simplify scheduling, and it sounded like harriesh might be doing it almost like a subroutine.