Recursive logic in Snaplogic

The pipeline was failing in production because it was calling the child pipeline for more than 64 times.
Do we have any workaround for this approach

Can you share more details on what you were trying to achieve, how many scheduled tasks you had due to which the parent pipeline was calling child for 64 times or share a high level design on why would a parent pipeline call child a couple of times.

Was was the frequency of calling child pipeline 64 times? A day, an hour, etc?
Thanks!

We don’t officially support recursion in the product. I think people generally work around this by changing the way they use the Pipeline Execute snap or triggered tasks.

upsertFolders_Marketo_Snowflake_2022_07_16.slp (25.6 KB)

I have attached the current pipeline where it goes to recursive more than 64 times. I would like to change the design for this and can someone provide how do i implement this?

My increment parameter is here offset and it needs to increment for every 200 records (0,199,398,597…) untill the warnings payload is not empty

Please find the attached get snap and the payload details

Once all the records are fetched i see the warning as No more assets to fetch

@sravankunadi - please check the REST Get snap documentation. The “Has Next” and “Next URL” properties will do what you are trying to do without recursion. The documentation also has examples to perform pagination, which is what you are doing here.

2 Likes

Thanks for your response @koryknick. Unfortunately we don’t have anything that we can rely on Has Next and Next URL.
Here is the response from marketo for marketo folders API:

The topic Connecting to Marketo with the REST Snap Pack describes pagination int eh REST Snaps with Marketo.

2 Likes

Thanks @dmiller. Pagination supports for activities but for the marketo folders there is no pagination that will support. Only the increment parameter needs to send as Offset untill it returns less than 200 this needs to go through loop

1 Like

@sravankunadi - what is the condition where you stop your recursion? That would be the same condition (or very similar) you would put in the Has Next property.

If I am reading the Marketo API specs correctly, you receive a “nextPageToken” when there are more results. So the Has Next would simply be the existence of “nextPageToken” element. Probably something like the following:

'nextPageToken' in $response

Sorry @koryknick if i am not clear on the requirement. We do not have anything nextPageToken for marketo folders extract and also we do not know when to terminate the process. if the records are less than 200 then we need to exit the job.

Marketo response on this:

We need to pass the offset as incremental parameter to fetch the records until the record count is less than 200. I have attached the pipeline in the above thread.

@sravankunadi - to make pagination a bit easier, I recommend that you build the complete URL, including the query parameters (except “offset”) in a Mapper just before the REST Get. This might take just a bit of testing to work out the correct syntax of the “root” element.

Then the REST Get properties could be set as follows:

  • Has Next: $entity.result.length == 200
  • Next URL: $original.url + “&offset=” + parseInt($entity.result.length) -1

Unfortunately, if the API doesn’t return a “next url”, you will need to build the full URL for the Next URL value.

Since I can’t call your service, I’m not able to replicate the folder pagination; and because I don’t have a Marketo instance with a large number of folders, I can’t provide a sample pipeline for you.

Hope this helps point you towards a solution!

1 Like

By the way - there is a Marketo snap pack that would let you easily list the Folder contents (as well as many other functions) - you may want to contact your SnapLogic CSM (Customer Success Manager) to ask about a trial in your dev org to see if this would work for your use case.

1 Like