Throttle Bot or Other Loop/Delay Mechanism?

[brent_vanallen]

I have a process where I am submitting a data load to an engine that reviews the data. I want to accomplish a couple things.

First - I want to be able to have the Pipeline pause for a delay I specify.

Second - I want the Pipeline to then “loop” and peridically check back using a DB lookup (Postgres Execute should work) to check the status of the submitted data…and do some kind of loop until it finds the status it wants, a timeout would be handy too in case it never finds the proper “complete” status.

Once it does find the right status it is looking for it would then continue past the loop condition and fire off post-processing steps in the database to clean data up, extract results of the submitted batch, etc.

I have seen a Snap referred to in the documentation as a “Throttle Bot Snap” in some examples (for instance this one https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2067890766/Use+Case+Employee+Onboarding), but I cannot actually find the snap itself or the Snap Pack it must come in. If anyone can help me with the above technical issue or tell me how to find the Throttle Bot Snap that woould be very appreciated.

Thank You,

Brent Van Allen
Aduro.

[rgoerwit]

I’ve given my boss, who is our SnapLogic admin, information needed for an enhancement request. I am new to SnapLogic, so I hesitate to critique. I’d just say that every other ETL tool and every other usable scripting or coding language I’ve used implements loops. So it’s a shocker that SL doesn’t have this - or that you have to use nested pipelines. So I wholeheartedly encourage them to look into this.

It’s very useful to have a visual that implements a loop and shows you, nicely, what the exit condition is and what variables/fields are getting modified as one iterates. It’s possible to use recursion (pipelines). But given that we can’t click through to child pipelines or see easily what’s getting passed to them (setup of paramaters is very clicky, on both the parent and child ends) - given all this, a loop construct is really needed.

It’s also important to improve recursion and make those parameters more visible. Having to click around the parent, and the child, and having to manually ensure that everything matches, is really hard to do. And recursion is difficult for the citizen-integrators that SL targets. It’s a hard concept. If SL wants to make this easy, a simple loop will help - along with visibility improvements, as noted.