Forum Discussion
Using the Group By N snap will work fine for smaller document sizes and document counts. Since grouping combines multiple documents and creates larger documents in memory, that approach is not recommended when document sizes or counts are large.
The batching option in the PipeExec snap does not support automatically passing a batch number. The parent pipeline can use an expression like ((snap.in.totalCount + 1000) / 1000).toFixed()
to generate a batch number to pass to the child pipeline, the child can use that info to generate the file name. See the attached parent and child pipelines
pparent_2022_03_03.slp (parent)
pchild_2022_03_03.slp (child)
A couple of releases ago we added a Memory Sensitivity
property to the Group By N
snap. Set it to Dynamic
to dynamically reduce the group size according to available memory conditions.
I am also facing same issue . Any update ? Have you solved it ?
Yes, we did solve it. The problem was that API Gateway resides in the public VPC portion of our AWS account, but the SnapLogic call was happening over our private direct connect tunnel into our local environment. We were able to solve the problem by enabling the endpoints as VPC link resources, and setting up a network load balancer that points to the internal SnapLogic URL.