Forum Discussion
HI salishrodinger ,
You can create a child pipeline in order to write the files into the external stage.
After all files are processed successfully, you can continue with the process.
Thank you marjan_karafilo for you reply, I did try back then to create a child pipeline right after the File Writer so it can process all my Snowflake executions. Parametered the child pipeline as a batch of 30 but somehow i did have some errors too
- arunsadhasivam6 years agoNew Contributor
no i tried json-splitter but the thinhg is it is returning collection of url , i need one url from collection at a time and use that url should be in rest post . let me know any option. i need some thing like queue or forloop. i tried sequence which can generate a counter 1-n records but i could not keep iterate over it.
i am new to snaplogic, i see only option is push to queue . but iam looking for some option other than this .- arunsadhasivam6 years agoNew Contributor
let me know if any option i missed in json splitter which could allow loop over collection one at a time.
The Splitter should be doing that for you. Can you maybe take a screenshot of your pipeline and a separate one of the Splitter configuration that you were trying at the time. We can take a look and try to figure out what is going on.
Note that when building a pipeline, you’re trying to setup the different stages of processing that will be applied to the data that is flowing through it. There’s not really any looping going on. So, in your case, you’re receiving the data, maybe validating it, and then doing the REST POST. The Splitter is needed to reshape the data flow to expand the single input doc into multiple output docs that can go through the remaining stages of the pipeline.