How to replicate the logic equivalent to while loop in Snaplogic pipelines

There is a child pipeline which returns values like completed, in queue, processing etc. The requirement is that parent pipeline to run this child pipeline till it returns Completed, of-course after certain wait time. Its like while loop, run the child pipeline, till the expression is true. How do accomplish this in snaplogic ?

Appreciate any pointers…

I’ve seen people suggest creating a Triggered Task for the child pipeline, and then have the parent pipeline call the child pipeline via REST GET snap and use the REST GET snap’s “Has Next” and “Next URL” capabilities (typically for paging) to do a while loop. The child pipeline has to return something that the parent “Has Next” can evaluate.

Can you explain what processing you’re doing? Maybe focus less on the mechanics so that we can understand the use case.

The parent pipeline queues up a bulk job to export records. The child pipeline keeps getting the status of the job every 60 seconds to get the status till it reaches its end state. I have tried to put everything in the parent pipeline but looks like child pipeline is the best way to go.

All these are REST API Get/POST jobs

What do you mean by “queue up” here? Is the queue in an external service and the children are pulling from there? Maybe I’m misunderstanding.

I guess I would expect the parent pipeline to pull from the queue and let the PipelineExecute snap distribute documents to child executions.

Sorry, I’m having trouble visualizing the parent and child pipelines here. Would you be able to attach a screenshot of them or an SLP export?

Yes. Its external (Marketo) service.Parent submits a bulk job and in the response object we get a exportID. We have to keep checking the status (REST Get Call to Marketo) of this export job thru another Marketo endpoint which happens in a child pipeline. This child pipeline should be running in while loop kind of thing till we get a success/failure response. Picture of child pipeline attached.

@MuraliNagula - This can be achieved using "Has next "and “Next URL” properties. Below rest snap invokes Next URL based on Has next flag. You can refer Snap documentation for more info.

I just wanted to add on that you should set the ‘Pagination interval’ as well so that the requests aren’t made too quickly.

Unfortunately the response object for this Marketo Endpoint doesn’t have “Has Next” or “Next URL” attributes. Otheriwse my life would have been much easier !!!

You can find the details of the response object etc in the following link.

https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Bulk_Export_Leads/getExportLeadsStatusUsingGET

GET /bulk/v1/leads/export/{exportId}/status.json

https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Bulk_Export_Leads/getExportLeadsStatusUsingGET

How and where do set these “Has Next” and “Next URL” in the triggered task ?

The “next” properties are expressions. So, I would think you could write an expression to check the response to the status request to see if it was completed or not. If it was completed, the ‘Has Next’ expression should return false, otherwise it should return true. The ‘Next URL’ can just be the same as the main Service URL property.

In the documentation for the REST Get Snap, there is an example for response form marketo:
REST Get Pagination: Marketo
For Marketo, set:

Has Next: resultdoc.moreResult
Next URL: “/rest/v1/activities.json?nextPageToken=” + resultdoc.nextPageToken + ”&activityTypeIds=1&activityTypeIds=12”

The difference is rest vs. bulk call.

The API for the child triggered task doesn’t have to return “Has Next” or “Next URL” properties directly, just something you can evaluate in expressions for those attributes in the REST GET snap.

For example:

2019-06-24_8-06-17

Or whatever type of expression that returns true/false based on whether you want the REST GET snap to do again…