Please find the attached sample pipelines:
pipe_1_2022_03_03.slp (6.7 KB) - parent
pipe_2_2022_03_03.slp (5.2 KB) - child
In the parent pipeline, using the Group By N snap you can specify the size of the group/batch to be sent to the child pipeline.
For every batch/group there will be separate child pipeline invocation and the number of the batch will be included in the filename.
Regards,
Spiro Taleski
we tried implementing this but it would cause an issue if we process more than 3 million records
Hi @Pretty,
Which field do you want to dynamically pass to the REST Get snap ?
@j.angelevski
This is URL passing in rest get snap
https://elastic.snaplogic.com/api/1/rest/public/runtime/org ?client name= comm@snaplogic.com
but I have n number of client name which we are passing in child pipeline
how can we call all the client name dynamically
