Using the Group By N snap will work fine for smaller document sizes and document counts. Since grouping combines multiple documents and creates larger documents in memory, that approach is not recommended when document sizes or counts are large.
The batching option in the PipeExec snap does not support automatically passing a batch number. The parent pipeline can use an expression like ((snap.in.totalCount + 1000) / 1000).toFixed() to generate a batch number to pass to the child pipeline, the child can use that info to generate the file name. See the attached parent and child pipelines
pparent_2022_03_03.slp (parent)
pchild_2022_03_03.slp (child)
A couple of releases ago we added a Memory Sensitivity property to the Group By N snap. Set it to Dynamic to dynamically reduce the group size according to available memory conditions.
Thank you. This worked well without impacting performance.
I’m glad to hear that! Thanks for the update.
