How to loop thru N times

[aakumar]

I have an input of 1000 Records coming in. I can only process 200 records at a time. How can I write a loop for this ?

[endor_force]

My first go-to would be to create  a sub pipeline for the part that needs to handle 200 records at a time and call that sub pipeline from the main pipeline using pipeline execute with the setting of batch size = 200.

Depending on what you need and your options, you may also be able to solve it by batching it in to groups of 200 using Group by N and processing each document as one batch.

[aakumar]

I tried the Group By N. Batch size 2.  But its not working. 

[aakumar]

I tried the Group By N. Batch size 2.  But its not working. 

[koryknick]

@aakumar - I think you misunderstood what @endor_force was proposing.  I think the suggestion to use Pipeline Execute with a "Batch size" setting of 200 is the right solution.  You can remove the Group By N in your parent pipeline with this option.  Basically, this will send groups of 200 records to a child pipeline and when that child has finished processing those 200 records, it will finish and as long as the parent more documents to process, it will continue to start child pipelines and send groups of 200 records.

Hope this helps!