โ12-07-2021 04:11 PM
I have a process where I am submitting a data load to an engine that reviews the data. I want to accomplish a couple things.
First - I want to be able to have the Pipeline pause for a delay I specify.
Second - I want the Pipeline to then โloopโ and peridically check back using a DB lookup (Postgres Execute should work) to check the status of the submitted dataโฆand do some kind of loop until it finds the status it wants, a timeout would be handy too in case it never finds the proper โcompleteโ status.
Once it does find the right status it is looking for it would then continue past the loop condition and fire off post-processing steps in the database to clean data up, extract results of the submitted batch, etc.
I have seen a Snap referred to in the documentation as a โThrottle Bot Snapโ in some examples (for instance this one https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2067890766/Use+Case+Employee+Onboarding), but I cannot actually find the snap itself or the Snap Pack it must come in. If anyone can help me with the above technical issue or tell me how to find the Throttle Bot Snap that woould be very appreciated.
Thank You,
Brent Van Allen
Aduro.
โ12-17-2021 09:56 AM
Thanks for the insight on the Snap. I used Viktorโs Ruby delay, but have been unable to get a loop working in order to (A) Submit a load to my MDM tool and (B) check back on DB status of the load to perform the next steps in the process. Setting this process up, Snaplogic looks like it avoids โenteringโ the loopโฆsticking on the step right before the loop starts.
I posted a diagram here, you can see the last Snap it will execute in greenโฆonce it gets here it just sits and spins. This shows that section of the pipeline where (A) we check back of data submitted upstream has finished processing, (B) we check back on that value and if not found to be โDONEโ yet we push the document stream back onto itself with a UNION (C).
โ12-17-2021 10:53 AM
Iโve done something like this before. I think youโre going to need a multi-pipeline design - I donโt believe there is a way to loop within a single pipeline. The below is just one high-level design idea - there are other options.
The step1 pipeline will do the initial work, then call step2 pipeline (using Pipeline Execute) which will have the delay and check status. The step2 pipeline can have a loopback-or-continue Router snap following the status check where if the status is complete, it forks to the logic to finish the process; but if status is not complete, then it recalls step1 pipeline with Pipeline Execute. You can use parameters and Router snap in the step1 pipeline to determine whether step1 skips the initial processing and simply recalls step2 again. Parameters can also be used as a counter mechanism, so that if the counter exceeds x-number retries, then step2 fails or notifies as desired.
The Pipeline Execute documentation states that it doesnโt support redundancy (even though it works in some of my attempts), otherwise, I would recommend a design within a single pipeline. But I stick with the documentation for pipeline longevity.
โ12-17-2021 03:03 PM
No luck with that approach :). Will have to just keep at it.
โ01-24-2022 02:45 PM
So did you come up with a decent solution?
BTW, any explanation for why SnapLogic doesnโt support a simple loop flow?
โ01-24-2022 03:55 PM
I was never able to get looping to work. Even with the โcallbackโ part of the loop separated out to a separate pipeline and using Pipeline Execute, it appears that Snaplogic can look ahead and detect if the logic in the datastream will create a potential loopโฆand refuses to enter the loop from the entry point (that would be looped back TO). I was able to get delay working via the suggested approach with a little Ruby script though.
We have not set it up yet but I anticipate that we will end up closing the loop logically by using a REST interface.