How to loop thru N times

Question

I have an input of 1000 Records coming in. I can only process 200 records at a time. How can I write a loop for this ?

endor_force · Answer

My first go-to would be to create&nbsp; a sub pipeline for the part that needs to handle 200 records at a time and call that sub pipeline from the main pipeline using pipeline execute with the setting of batch size = 200.Depending on what you need and your options, you may also be able to solve it by batching it in to groups of 200 using Group by N and processing each document as one batch.

aakumar · Answer

I tried the Group By N. Batch size 2.&nbsp; But its not working.&nbsp;

aakumar · Answer

I tried the Group By N. Batch size 2.&nbsp; But its not working.&nbsp;

koryknick · Answer

aakumar&nbsp;- I think you misunderstood what&nbsp;endor_force&nbsp;was proposing.&nbsp; I think the suggestion to use Pipeline Execute with a "Batch size" setting of 200 is the right solution.&nbsp; You can remove the Group By N in your parent pipeline with this option.&nbsp; Basically, this will send groups of 200 records to a child pipeline and when that child has finished processing those 200 records, it will finish and as long as the parent more documents to process, it will continue to start child pipelines and send groups of 200 records.
Hope this helps!

Forum Discussion

How to loop thru N times

Recent Discussions

Pagination and nextCursor in header

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?