Creating and testing error pipelines

Designing Pipelines
#1

I’m trying to create an error pipeline. A couple of questions…

  • Is it supposed to be called regardless of whether there is an error or not? (It is. Am I doing something wrong? or is this expected behavior) If, so, how should I determine whether there is an error, or not, within the error pipeline?

  • Is there any way I can trigger various errors for testing purposes? So I can test that the error pipeline will respond correctly for different problems?

Most of the serious problems we need to recover from involve the temporary inability to connect to various sites to get or put files, and I can’t figure out a way to fake those errors.

#2

Hello @ wpenfold

  1. No, error pipeline is called only when some error occurs. You are doing something wrong.

  2. You can mimic that error , trying to connect to that same site with Rest Get snap( while the site is unresponsive). That way the error will be thrown to the error pipeline.

Best Regards
Dimche Saveski

#3

My pipeline is finishing successfully, but the error pipeline is still being called. What could I be doing wrong to make this happen?

I can’t make the vendor sites unresponsive when I want them to be. If sftp-ing a file to a vendor site fails, this is a serious issue I need to be able to handle,
but I have no idea what data will presumably get sent to the error pipeline when that happens, and I am unable to test a solution.

#4

If you have an error pipeline defined for your base pipeline, then I believe it will always run (and wait for error data) when your base pipeline runs, not only run when there is actually an error. If you have an error path on a snap, then it should only have data passed that way when errors occur.

2 Likes
#5

Dear wpenfold

You can see which record caused the error, what was the reason for that error, which snap failed and e.t.c in the error pipeline.
You can check this error handling article, where error handling is explained in great details.

Best Regards
Dimche Saveski

2 Likes
#6

That is a very good article, thanks. That’s pretty much what I’m trying to do with my error pipeline, get the error, write to an error log in the database, determine who to email the issue to, and if possible, take corrective action. However, I still don’t know how the specific error data is going to look–ie, what error message will I get if the sftp to the vendor site fails so I can have the error pipeline perform the appropriate actions? And how do I test the functioning of the error pipeline before I put it in production? Is there a table somewhere, of all the errors that snaplogic might return? Is there a way to fake the errors for testing purposes?

#7

1 way of doing it would be:
Making simple error pipeline just from 3 snaps, open Mapper with error, reason and resolution fields in the input schema, downstream JSON Formater and downstream File Writer.
Then feed the source(Json Generator just or testing purposes) with fault records( send records without auth, records with fault target url, records with non existing fields and e.t.c ) , and checking created file with File Writer, for more detailed explanation on errors.
Regards

1 Like
#8

We do much the same thing here. We insert a row into an Event Log table in the database. It contains a timestamp, some other bookkeeping information, a message string, and a payload. We just take the entirely of the JSON error stream and dump it into that payload column. Other stuff gets parsed and nicely formatted, but having that payload column helps when we get errors that we really weren’t expecting.

We just copy a payload out to a JSON editor which formats and validates it, and then we can understand more or less what went wrong.

By the way @wpenfold, I can confirm, the error pipeline that catches all errors for your processing pipeline does, in fact, “spool up” every time a pipeline that uses it starts running. At my shop, we look at the Snaplogic pipeline logs for two things: obviously if a pipeline message line is red and indicates an error. But the second thing we check is, “does the error pipeline show a little plus-sign-box [+] next to it?” If there is no box, the error pipeline had nothing to report. If the box is there, we click on it and it expands out to show the conditions that the error pipeline caught.

Hope that helps!!

1 Like
#9

Is there a table of errors that Snaplogic might throw? It sounds like the only way I can find out what the errors might be, is put this in production and wait for a failure? I can’t make the vendor sites fail. Hey VendorX, can you please pull the plug on your server for 15 minutes while I test? I’m sure your other 1000 clients won’t mind. LOL

#10

An “error pipeline” might not be right for you if you want specific processing for various types of errors that various snaps can give. In my opinion this would be for more generic error handling, and there are usually pretty standard fields ($error, $reason, $error_entity) that you can throw to logs or whatever for troubleshooting.

For specific error handling on different snaps within your pipeline, you might want to put the error processing on the snap’s error path itself. For example, “pull the pug” (can’t connect) type errors, you should be able to test your sad/error paths by trying to hit something else (bad url, etc.) rather than asking your vendor to kill their site… I’m sure you can figure out how to test your code without the participation of your partners.

1 Like
#11

Well, good point. Currently I don’t have that option—accounts are set up by the snaplogic administrator and firewalls by security team, but I guess I could see if they’ll set up a fake one for testing

#12

I have not seen a table of all errors, but that would be useful. Perhaps someone at Snaplogic will reach out and comment?

I know they practice some variant of Agile, so there may not be a central table of errors to be referred to; errors may be generated at the points where the code can recognize a problem, and it might be constantly in flux with continuing development.

#13

There currently is not one central table of errors. Some you might see are common HTTP responses, others may be coming in from the applications that you are integrating, etc. I have forwarded this to our Doc team for messages specific to our platform.