Error pipelines

Harish:

I’ll post some tips here that we’ve learned on our own, before I read Diane Miller’s links. (Diane’s suggestions are usually more robust and more helpful than my own, so where the two sets of tips may conflict, do what she suggests!)

1. First and foremost, error pipelines must never be able to produce an error themselves. Make them bulletproof.
2. Consider making the error pipeline log an error somewhere. A REST call to another system is best: that way your log is immune to local crashes. Other good options are to a local lightweight RDBMS table, a remote DB, or using a library that implements rotating text files.
3. General error handlers are better for doing error reporting. You can also build specific error handlers for situations where it may be possible to recover from an error. (These are not the global “exception catcher” type you attach to the overall pipeline. These are the ones you attach to the “error periscope” outputs.)
4. Have two variants of your general error handler: one that logs the error, then halts everything and throws a big red error on the SnapLogic dashboard, and one that silently eats the error but logs it. Some situations call for one; some for the other.

Hope this helps!