Forum Discussion
we tried implementing this but it would cause an issue if we process more than 3 million records
Using the Group By N snap will work fine for smaller document sizes and document counts. Since grouping combines multiple documents and creates larger documents in memory, that approach is not recommended when document sizes or counts are large.
The batching option in the PipeExec snap does not support automatically passing a batch number. The parent pipeline can use an expression like ((snap.in.totalCount + 1000) / 1000).toFixed()
to generate a batch number to pass to the child pipeline, the child can use that info to generate the file name. See the attached parent and child pipelines
pparent_2022_03_03.slp (parent)
pchild_2022_03_03.slp (child)
- eric_sou7 years agoNew Contributor III
So nobody would have an idea how to quickly generate data?