Forum Discussion
koryknick
6 months agoEmployee
SL12345 - You can also use the Pipeline Execute snap and set the Batch Size value to your desired number of records in the target. Create a child pipeline that is just a CSV Formatter and File Writer snap and call that child in your Pipeline Execute.
Basically, this passes the number of records specified in the Batch Size property to an instance of the child pipeline which can create a file of those records and finish, then the parent will start a new child with the next "batch" of records, and will keep iterating until all input records are consumed. Simple data chunking of your original file.
Hope this helps!