Creating and testing error pipelines
I’m trying to create an error pipeline. A couple of questions…
Is it supposed to be called regardless of whether there is an error or not? (It is. Am I doing something wrong? or is this expec...
Forum Discussion
That is a very good article, thanks. That’s pretty much what I’m trying to do with my error pipeline, get the error, write to an error log in the database, determine who to email the issue to, and if possible, take corrective action. However, I still don’t know how the specific error data is going to look–ie, what error message will I get if the sftp to the vendor site fails so I can have the error pipeline perform the appropriate actions? And how do I test the functioning of the error pipeline before I put it in production? Is there a table somewhere, of all the errors that snaplogic might return? Is there a way to fake the errors for testing purposes?
1 way of doing it would be:
Making simple error pipeline just from 3 snaps, open Mapper with error, reason and resolution fields in the input schema, downstream JSON Formater and downstream File Writer.
Then feed the source(Json Generator just or testing purposes) with fault records( send records without auth, records with fault target url, records with non existing fields and e.t.c ) , and checking created file with File Writer, for more detailed explanation on errors.
RegardsWe do much the same thing here. We insert a row into an Event Log table in the database. It contains a timestamp, some other bookkeeping information, a message string, and a payload. We just take the entirely of the JSON error stream and dump it into that payload column. Other stuff gets parsed and nicely formatted, but having that payload column helps when we get errors that we really weren’t expecting.
We just copy a payload out to a JSON editor which formats and validates it, and then we can understand more or less what went wrong.
By the way @wpenfold, I can confirm, the error pipeline that catches all errors for your processing pipeline does, in fact, “spool up” every time a pipeline that uses it starts running. At my shop, we look at the Snaplogic pipeline logs for two things: obviously if a pipeline message line is red and indicates an error. But the second thing we check is, “does the error pipeline show a little plus-sign-box [+] next to it?” If there is no box, the error pipeline had nothing to report. If the box is there, we click on it and it expands out to show the conditions that the error pipeline caught.
Hope that helps!!
Is there a table of errors that Snaplogic might throw? It sounds like the only way I can find out what the errors might be, is put this in production and wait for a failure? I can’t make the vendor sites fail. Hey VendorX, can you please pull the plug on your server for 15 minutes while I test? I’m sure your other 1000 clients won’t mind. LOL
