Forum Discussion
Using the Group By N snap will work fine for smaller document sizes and document counts. Since grouping combines multiple documents and creates larger documents in memory, that approach is not recommended when document sizes or counts are large.
The batching option in the PipeExec snap does not support automatically passing a batch number. The parent pipeline can use an expression like ((snap.in.totalCount + 1000) / 1000).toFixed()
to generate a batch number to pass to the child pipeline, the child can use that info to generate the file name. See the attached parent and child pipelines
pparent_2022_03_03.slp (parent)
pchild_2022_03_03.slp (child)
A couple of releases ago we added a Memory Sensitivity
property to the Group By N
snap. Set it to Dynamic
to dynamically reduce the group size according to available memory conditions.
- gowdhaman0085 years agoNew Contributor II
Take a look at the following script. You should use the file format or document to binary snap. Zip write accepts many files can be written here. Take a look at the following screenshot, your each process should connect to formater / document to binary and send to zip write.
Hope this helps.!!
Hi Gowtham,
Thanks for the reply , indeed my snap does not contain the output data in json format also instead of writing multiple files at the end is there any way of including all those excels commonly under a zip . If yes, that could be really helpful.
Thanks,
- gowdhaman0085 years agoNew Contributor II
You can try something similar as follows.I was trying to show you some sudo code on how to do that. You can refer to the following screenshot, it may give you some idea as your pipe is having many SOQL query and excel formatting.
Thanks,
Gowdhaman
Hi Gowtham,
I have tried the same but the output in the zip we don’t have the excel files instead the files are encrypted. Also please find my updated SLP.
CALL PLAN RELATED VALIDATION_2020_11_03.slp (168.4 KB)
Related Content
- 10 months ago
- 2 years ago
- 3 years ago