Pipeline execute maximum recursion calls depth

Question

Hi Team,
I have designed pipeline to make recursive calls using pipeline execute snap…in one scenario i got an error at the 64th iteration tat it ‘‘cannot execute any more child pipelines’’ and i found in pipeline execute snap documentation that it cannot exceed more than 64 depth b4 they begin failing…
But wen i executed the same pipeline after a week…its taking 150 iterations…how is this possible…

koryknick · Answer

@Noushin - to my knowledge, 64 is still the limit.  I just created the attached pipeline to test the depth and 64 levels completes successfully but 65 fails.  This was tested in 2 different orgs, so I’m fairly confident that this threshold has not changed.
Test Recursion_2022_04_15.slp (9.9 KB)

noushin · Answer

Hi @koryknick - Please find the below attached screenshots where both error at 64th iteration and also the iterations that have reached 99 is attached

second image has gone until 99th iteration…

koryknick · Answer

@Noushin - let’s take a step back and make sure we are both using the correct terminology.
Recursion would be a pipeline that calls itself via Pipeline Execute - maximum depth is 64 calls deep in the stack of pipelines.  Recursion is typically used to perform a task based on the current input value until a condition is met.  For example, you may need to use recursion to get the full directory tree of a file system, so you get the listing of the current directory and then recursively call the pipeline for each directory to get the file list of each child directory.   Recursion in Dashboard looks like this:

Iteration is calling a child pipeline by passing input documents in the stream to perform operations on each document.  This is the typical (and recommended) pattern of calling the Pipeline Execute.  Iteration in Dashboard looks like this:

Based on the error I see in your original thread, I believe you mean that previous executions only allowed 64 iterations (not recursion) and would indicate that something was consuming too many resources on the Snaplex node to allow any more child pipelines to execute.  You may want to work with your SnapLogic CSM to look at your capacity planning.  There is no limit to the number of child pipeline executions but you would lose visibility of the statistics after a number of iterations.

noushin · Answer

Hi @koryknick - Even I am making recursive calls using pipeline execute snap with condition until all records from the soap services are extracted i call same pipeline in recursive mode…And in the image i have dedelected the group pipelines checkbox hence its not on tree structure, please find the below image which depicts the recursive call.

noushin · Answer

Hi @koryknick - My scenario is I have to load all the data from the given data source. The maximum records it can send me in one iteration is 50k(its configurable)
I have designed pipeline in such a way,
pipeline XYZ first invokes the backend using soap execute snap, which returns 50k records in first iteration and I have used router snap to check the condition if the record count is less than 50k then i stop the pipeline else I invoke same pipeline XYZ within itself …(XYZ calling XYZ) until the condition is met…
This recursive invocation ryt…

Forum Discussion

Pipeline execute maximum recursion calls depth

Recent Discussions

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?

Multipart Reader failure - 'content-type' was not found