SnapLogic - Integration Nation

Harriesh · ‎02-28-2018

How to wait child pipeline till parent pipelines are completed. Currently the child pipelines are getting executed before the parent execution done with all the documents.
Because of that it is having memory issue.

Is there any wait to wait the child pipeline till all the documents execution and then call the child pipeline.

del · ‎03-06-2018

@Harriesh, you might look at using the Group By N snap with Group Size of 0 at the end of the parent pipeline to aggregate all the documents into one and then start the Child pipeline with a splitter to break it back down into individual documents. I don’t know how much that would help out memory-wise, but I suspect it might be better than a sort.

View solution in original post

stephenknilans · ‎02-28-2018

First of all, don’t have preceding tasks send any output to following tasks, unless it is the end of their process. If you do, the following tasks(which would deal with the child records), will run. Of course, you COULD provide aggregates of keys and accept that data to join in the child. This allows feedback, aggregation, RI validation, and processes children only after the parents are processed. Of course there is no value to that other than to isolate processes, and allow distributing among multiple nodes. OH, and doing that ALSO limits memory consumption if you use multiple JVMs, or multiple nodes.

Harriesh · ‎03-01-2018

Let me try out that option…

Thanks.

aleung · ‎03-01-2018

You could also try writing the output to a file / db from the parent pipeline, that way you can force all processing is done prior to initiate the child.

stephenknilans · ‎03-02-2018

I probably should have said that. I have one case where I am doing precisely that. But I am kicking off the tasks to handle parameters and order, and simplify scheduling, and it sounded like harriesh might be doing it almost like a subroutine.