Can you explain what processing you’re doing? Maybe focus less on the mechanics so that we can understand the use case.
The parent pipeline queues up a bulk job to export records. The child pipeline keeps getting the status of the job every 60 seconds to get the status till it reaches its end state. I have tried to put everything in the parent pipeline but looks like child pipeline is the best way to go.
All these are REST API Get/POST jobs
What do you mean by “queue up” here? Is the queue in an external service and the children are pulling from there? Maybe I’m misunderstanding.
I guess I would expect the parent pipeline to pull from the queue and let the PipelineExecute snap distribute documents to child executions.
Sorry, I’m having trouble visualizing the parent and child pipelines here. Would you be able to attach a screenshot of them or an SLP export?
Yes. Its external (Marketo) service.Parent submits a bulk job and in the response object we get a exportID. We have to keep checking the status (REST Get Call to Marketo) of this export job thru another Marketo endpoint which happens in a child pipeline. This child pipeline should be running in while loop kind of thing till we get a success/failure response. Picture of child pipeline attached.
