Throttle Bot or Other Loop/Delay Mechanism?

[brent_vanallen]

I have a process where I am submitting a data load to an engine that reviews the data. I want to accomplish a couple things.

First - I want to be able to have the Pipeline pause for a delay I specify.

Second - I want the Pipeline to then “loop” and peridically check back using a DB lookup (Postgres Execute should work) to check the status of the submitted data…and do some kind of loop until it finds the status it wants, a timeout would be handy too in case it never finds the proper “complete” status.

Once it does find the right status it is looking for it would then continue past the loop condition and fire off post-processing steps in the database to clean data up, extract results of the submitted batch, etc.

I have seen a Snap referred to in the documentation as a “Throttle Bot Snap” in some examples (for instance this one https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2067890766/Use+Case+Employee+Onboarding), but I cannot actually find the snap itself or the Snap Pack it must come in. If anyone can help me with the above technical issue or tell me how to find the Throttle Bot Snap that woould be very appreciated.

Thank You,

Brent Van Allen
Aduro.

[viktor_n]

Hi @brent.vanallen,

Regardless delaying, you can accomplish that with Script snap.
Here I use python

• At the top first import time module
• Use time.sleep(sec) method

from com.snaplogic.scripting.language import ScriptHook
import time 

class TransformScript(ScriptHook):
    def __init__(self, input, output, error, log):
        self.input = input
        self.output = output
        self.error = error
        self.log = log

    def execute(self):
        self.log.info("Executing Transform script")

        ## Move time.sleep() here out of while loop if you want to wait only by the seconds you have specified not more.

        while self.input.hasNext():
            try:
                inDoc = self.input.next()

                ## Here will wait for each document. That means if we have 5 documents at input and sleep is set for 10 seconds, it will wait 50 seconds till returns the output.
                ## In parameter specify the time in seconds. You can make it time dynamic if you specify the time out of the script snap and after that by getting the value with inDoc.get("field_Name")
                time.sleep(10) 

                self.output.write(inDoc, inDoc)
            except Exception as e:
                errDoc = {
                    'error' : str(e)
                }
                self.log.error("Error in python script")
                self.error.write(errDoc)

        self.log.info("Script executed")

    def cleanup(self):
        self.log.info("Cleaning up")

hook = TransformScript(input, output, error, log)

Regards,
Viktor

[SpiroTaleski]

@brent.vanallen

Regarding the delaying, the Python script provided by @viktor_n can be used. I am not aware of the Throttle Bot Snap as well.

Regarding the loop mechanism. Maybe is better to avoid any kind of loops and try with a process that will be scheduled(using scheduled task) to run on some predestined time interval and will do the DB lookup. If there is no desired status found then simply complete the execution, and if there is a status found then execute the post-processing steps.

Regards,
Spiro Taleski

[brent_vanallen]

@Spiro_Taleski, I can see this has been a feature request for 4 years based on the link you gave…also looks like I am not alone at all in this request. I am surprised that Snaplogic can’t just create a simple Snap to accomplish this given the desire in the community to have it.

[mbowen]

Hi @brent.vanallen

Throttle Bot is a snap in a “test” snap pack that we use internally. It probably shouldn’t have been referenced in our documentation.

We do have an Snap SDK which allow customers to create their own custom snaps. Now, I’m not necessarily suggesting this for this case, and get it, that this should probably be an available snap.

Thank you @Spiro_Taleski and @viktor_n for all of your contributions to community. I’ve actually learned quite a bit from your answers.

I’ll ask the Snap team regarding whether we would consider making one or more of these “test” snaps available. These snaps are sufficient for our usage, but may need a bit of work before being released to a wider audience which is probably why they haven’t.