Pipeline Execute Pool size By: Kareem Salem I am trying to use the pipeline execute snap to launch child pipelines. currently the throughput of this is relatively slow, to improve this I would like to increase the pool size to allow multiple childs to run at the same time however regardless of changing this setting it seems the run continues to only process one document at a time. for example please see screenshots below: In the screenshots i would expect 10 documents to be on the input section, with 10 instances of the DELME pipeline running, however only 1 is running at a time?
If you look in Monitor at the list of child pipelines executing under your parent pipeline, you will likely see that there are indeed 10 concurrent pipelines running - as one completes, a new one starts - so the start/end times of the pipelines will overlap. The reason you see only one document difference between the input/output of the child pipeline execution is because as soon as one child completes, the next is started. Here is a screenshot of an execution I just did that scans 24 files, then sends to a child for processing, which takes about 3 seconds per file. You can see that the last 4 files all started at 10:26:09, the previous 10 started at 10:26:06, and the first 10 had started at 10:26:02 Hopefully this helps clarify!
