brent_vanallen
New Contributor
Joined 4 years ago
User Profile
User Widgets
Contributions
Re: Throttle Bot or Other Loop/Delay Mechanism?
I was never able to get looping to work. Even with the “callback” part of the loop separated out to a separate pipeline and using Pipeline Execute, it appears that Snaplogic can look ahead and detect if the logic in the datastream will create a potential loop…and refuses to enter the loop from the entry point (that would be looped back TO). I was able to get delay working via the suggested approach with a little Ruby script though. We have not set it up yet but I anticipate that we will end up closing the loop logically by using a REST interface.1.7KViews0likes1CommentRe: Throttle Bot or Other Loop/Delay Mechanism?
Thanks for the insight on the Snap. I used Viktor’s Ruby delay, but have been unable to get a loop working in order to (A) Submit a load to my MDM tool and (B) check back on DB status of the load to perform the next steps in the process. Setting this process up, Snaplogic looks like it avoids “entering” the loop…sticking on the step right before the loop starts. I posted a diagram here, you can see the last Snap it will execute in green…once it gets here it just sits and spins. This shows that section of the pipeline where (A) we check back of data submitted upstream has finished processing, (B) we check back on that value and if not found to be “DONE” yet we push the document stream back onto itself with a UNION (C).1.7KViews0likes5CommentsRe: Throttle Bot or Other Loop/Delay Mechanism?
@Spiro_Taleski, I can see this has been a feature request for 4 years based on the link you gave…also looks like I am not alone at all in this request. I am surprised that Snaplogic can’t just create a simple Snap to accomplish this given the desire in the community to have it.3.3KViews0likes7CommentsThrottle Bot or Other Loop/Delay Mechanism?
I have a process where I am submitting a data load to an engine that reviews the data. I want to accomplish a couple things. First - I want to be able to have the Pipeline pause for a delay I specify. Second - I want the Pipeline to then “loop” and peridically check back using a DB lookup (Postgres Execute should work) to check the status of the submitted data…and do some kind of loop until it finds the status it wants, a timeout would be handy too in case it never finds the proper “complete” status. Once it does find the right status it is looking for it would then continue past the loop condition and fire off post-processing steps in the database to clean data up, extract results of the submitted batch, etc. I have seen a Snap referred to in the documentation as a “Throttle Bot Snap” in some examples (for instance this one https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2067890766/Use+Case+Employee+Onboarding), but I cannot actually find the snap itself or the Snap Pack it must come in. If anyone can help me with the above technical issue or tell me how to find the Throttle Bot Snap that woould be very appreciated. Thank You, Brent Van Allen Aduro.7.1KViews0likes10CommentsChallenges With Slow Match Snap Levenshtein Comparisons
We are using the Match Snap to do fuzzy-comparisons and match Customer records without a unique primary key. This particular Snap is in the ML Data Preparation Snap pack. We traced a bottleneck to uses of the Match Snap that are using the Levenshtein comparison method, and that makes sense as this algorithm is expensive. Has anyone found any techniques for making Levenshtein comparisons faster like record set binning, pre-sort, or other methods can could make it faster than the just-out-of-the-box usage? I could potentially use direct calls to Postgres to use the native Levelshtein distance functions but I doubt that would be better and probably worse. Thanks Much, Brent Van Allen Aduro, LLC. Keywords: Match Performance, Levenshtein, Fuzzy Matching, Data Prep, Machine Learning1.3KViews0likes0Comments