Creating and testing error pipelines

[wpenfold]

I’m trying to create an error pipeline. A couple of questions…

• Is it supposed to be called regardless of whether there is an error or not? (It is. Am I doing something wrong? or is this expected behavior) If, so, how should I determine whether there is an error, or not, within the error pipeline?

• Is there any way I can trigger various errors for testing purposes? So I can test that the error pipeline will respond correctly for different problems?

Most of the serious problems we need to recover from involve the temporary inability to connect to various sites to get or put files, and I can’t figure out a way to fake those errors.

[wpenfold]

That is a very good article, thanks. That’s pretty much what I’m trying to do with my error pipeline, get the error, write to an error log in the database, determine who to email the issue to, and if possible, take corrective action. However, I still don’t know how the specific error data is going to look–ie, what error message will I get if the sftp to the vendor site fails so I can have the error pipeline perform the appropriate actions? And how do I test the functioning of the error pipeline before I put it in production? Is there a table somewhere, of all the errors that snaplogic might return? Is there a way to fake the errors for testing purposes?

[dimche_saveski]

1 way of doing it would be:
Making simple error pipeline just from 3 snaps, open Mapper with error, reason and resolution fields in the input schema, downstream JSON Formater and downstream File Writer.
Then feed the source(Json Generator just or testing purposes) with fault records( send records without auth, records with fault target url, records with non existing fields and e.t.c ) , and checking created file with File Writer, for more detailed explanation on errors.
Regards

[ForbinCSD]

We do much the same thing here. We insert a row into an Event Log table in the database. It contains a timestamp, some other bookkeeping information, a message string, and a payload. We just take the entirely of the JSON error stream and dump it into that payload column. Other stuff gets parsed and nicely formatted, but having that payload column helps when we get errors that we really weren’t expecting.

We just copy a payload out to a JSON editor which formats and validates it, and then we can understand more or less what went wrong.

By the way @wpenfold, I can confirm, the error pipeline that catches all errors for your processing pipeline does, in fact, “spool up” every time a pipeline that uses it starts running. At my shop, we look at the Snaplogic pipeline logs for two things: obviously if a pipeline message line is red and indicates an error. But the second thing we check is, “does the error pipeline show a little plus-sign-box [+] next to it?” If there is no box, the error pipeline had nothing to report. If the box is there, we click on it and it expands out to show the conditions that the error pipeline caught.

Hope that helps!!

[wpenfold]

Is there a table of errors that Snaplogic might throw? It sounds like the only way I can find out what the errors might be, is put this in production and wait for a failure? I can’t make the vendor sites fail. Hey VendorX, can you please pull the plug on your server for 15 minutes while I test? I’m sure your other 1000 clients won’t mind. LOL

[ForbinCSD]

I have not seen a table of all errors, but that would be useful. Perhaps someone at Snaplogic will reach out and comment?

I know they practice some variant of Agile, so there may not be a central table of errors to be referred to; errors may be generated at the points where the code can recognize a problem, and it might be constantly in flux with continuing development.