I’m trying to create an error pipeline. A couple of questions… Is it supposed to be called regardless of whether there is an error or not? (It is. Am I doing something wrong? or is this expected behavior) If, so, how should I determine whether there is an error, or not, within the error pipeline? Is there any way I can trigger various errors for testing purposes? So I can test that the error pipeline will respond correctly for different problems? Most of the serious problems we need to recover from involve the temporary inability to connect to various sites to get or put files, and I can’t figure out a way to fake those errors.

If you have an error pipeline defined for your base pipeline, then I believe it will always run (and wait for error data) when your base pipeline runs, not only run when there is actually an error. If you have an error path on a snap, then it should only have data passed that way when errors occur.

wpenfold: what data will presumably get sent to t Dear wpenfold You can see which record caused the error, what was the reason for that error, which snap failed and e.t.c in the error pipeline. You can check this error handling article, where error handling is explained in great details. Best Regards Dimche Saveski

1 way of doing it would be: Making simple error pipeline just from 3 snaps, open Mapper with error, reason and resolution fields in the input schema, downstream JSON Formater and downstream File Writer. Then feed the source(Json Generator just or testing purposes) with fault records( send records without auth, records with fault target url, records with non existing fields and e.t.c ) , and checking created file with File Writer, for more detailed explanation on errors. Regards

We do much the same thing here. We insert a row into an Event Log table in the database. It contains a timestamp, some other bookkeeping information, a message string, and a payload. We just take the entirely of the JSON error stream and dump it into that payload column. Other stuff gets parsed and nicely formatted, but having that payload column helps when we get errors that we really weren’t expecting. We just copy a payload out to a JSON editor which formats and validates it, and then we can understand more or less what went wrong. By the way @wpenfold, I can confirm, the error pipeline that catches all errors for your processing pipeline does, in fact, “spool up” every time a pipeline that uses it starts running. At my shop, we look at the Snaplogic pipeline logs for two things: obviously if a pipeline message line is red and indicates an error. But the second thing we check is, “does the error pipeline show a little plus-sign-box [+] next to it?” If there is no box, the error pipeline had nothing to report. If the box is there, we click on it and it expands out to show the conditions that the error pipeline caught. Hope that helps!!

An “error pipeline” might not be right for you if you want specific processing for various types of errors that various snaps can give. In my opinion this would be for more generic error handling, and there are usually pretty standard fields ($error, $reason, $error_entity) that you can throw to logs or whatever for troubleshooting. For specific error handling on different snaps within your pipeline, you might want to put the error processing on the snap’s error path itself. For example, “pull the pug” (can’t connect) type errors, you should be able to test your sad/error paths by trying to hit something else (bad url, etc.) rather than asking your vendor to kill their site… I’m sure you can figure out how to test your code without the participation of your partners.

Creating and testing error pipelines | SnapLogic - Integration Nation

12 Replies

christwr
Contributor III
6 years ago
If you have an error pipeline defined for your base pipeline, then I believe it will always run (and wait for error data) when your base pipeline runs, not only run when there is actually an error. If you have an error path on a snap, then it should only have data passed that way when errors occur.
christwr
Contributor III
6 years ago
An “error pipeline” might not be right for you if you want specific processing for various types of errors that various snaps can give. In my opinion this would be for more generic error handling, and there are usually pretty standard fields ($error, $reason, $error_entity) that you can throw to logs or whatever for troubleshooting.

For specific error handling on different snaps within your pipeline, you might want to put the error processing on the snap’s error path itself. For example, “pull the pug” (can’t connect) type errors, you should be able to test your sad/error paths by trying to hit something else (bad url, etc.) rather than asking your vendor to kill their site… I’m sure you can figure out how to test your code without the participation of your partners.
- wpenfold
  Contributor
  6 years ago
  Well, good point. Currently I don’t have that option—accounts are set up by the snaplogic administrator and firewalls by security team, but I guess I could see if they’ll set up a fake one for testing
dimche_saveski
New Contributor III
6 years ago
Hello @ wpenfold

No, error pipeline is called only when some error occurs. You are doing something wrong.

You can mimic that error , trying to connect to that same site with Rest Get snap( while the site is unresponsive). That way the error will be thrown to the error pipeline.

Best Regards
Dimche Saveski
- wpenfold
  Contributor
  6 years ago
  My pipeline is finishing successfully, but the error pipeline is still being called. What could I be doing wrong to make this happen?
  
  I can’t make the vendor sites unresponsive when I want them to be. If sftp-ing a file to a vendor site fails, this is a serious issue I need to be able to handle,
  but I have no idea what data will presumably get sent to the error pipeline when that happens, and I am unable to test a solution.
  - dimche_saveski
    New Contributor III
    6 years ago
    
    wpenfold:
    
    what data will presumably get sent to t
    
    Dear wpenfold
    
    You can see which record caused the error, what was the reason for that error, which snap failed and e.t.c in the error pipeline.
    You can check this error handling article, where error handling is explained in great details.
    
    Best Regards
    Dimche Saveski
wpenfold
Contributor
6 years ago
That is a very good article, thanks. That’s pretty much what I’m trying to do with my error pipeline, get the error, write to an error log in the database, determine who to email the issue to, and if possible, take corrective action. However, I still don’t know how the specific error data is going to look–ie, what error message will I get if the sftp to the vendor site fails so I can have the error pipeline perform the appropriate actions? And how do I test the functioning of the error pipeline before I put it in production? Is there a table somewhere, of all the errors that snaplogic might return? Is there a way to fake the errors for testing purposes?
- dimche_saveski
  New Contributor III
  6 years ago
  1 way of doing it would be:
  Making simple error pipeline just from 3 snaps, open Mapper with error, reason and resolution fields in the input schema, downstream JSON Formater and downstream File Writer.
  Then feed the source(Json Generator just or testing purposes) with fault records( send records without auth, records with fault target url, records with non existing fields and e.t.c ) , and checking created file with File Writer, for more detailed explanation on errors.
  Regards
  - ForbinCSD
    Contributor
    6 years ago
    We do much the same thing here. We insert a row into an Event Log table in the database. It contains a timestamp, some other bookkeeping information, a message string, and a payload. We just take the entirely of the JSON error stream and dump it into that payload column. Other stuff gets parsed and nicely formatted, but having that payload column helps when we get errors that we really weren’t expecting.
    
    We just copy a payload out to a JSON editor which formats and validates it, and then we can understand more or less what went wrong.
    
    By the way @wpenfold, I can confirm, the error pipeline that catches all errors for your processing pipeline does, in fact, “spool up” every time a pipeline that uses it starts running. At my shop, we look at the Snaplogic pipeline logs for two things: obviously if a pipeline message line is red and indicates an error. But the second thing we check is, “does the error pipeline show a little plus-sign-box [+] next to it?” If there is no box, the error pipeline had nothing to report. If the box is there, we click on it and it expands out to show the conditions that the error pipeline caught.
    
    Hope that helps!!
dmiller
Former Employee
6 years ago
There currently is not one central table of errors. Some you might see are common HTTP responses, others may be coming in from the applications that you are integrating, etc. I have forwarded this to our Doc team for messages specific to our platform.