Using "Error Pipeline" - parent pipeline does not stop, how to stop it? error-pipeline called at the end?

[Szymon]

Hi SL gurus!

Pipeline got Error Pipeline defined in properties(for logging/alerting).
When error occures, in e.g. Mapper, main pipeline does not stop and execute error-pipeline, but runs further. If more errors occur, they will be sent to the error-pipeline as an array.
What is more, in Dashboard, main pipeline gets status Completed, not Failed.
Is there a possibility to use error-pipeline, so error logging is easier and stop the pipeline, when an error occurs?

thanks
Simon 🙂 😄

[jcampion]

Hi @Szymon. Were you ever able to solve this? I am seeing the same behavior and would appreciate any advice. Thanks!

[cjhoward18]

@jcampion @Szymon

If you’d like the error pipeline to also throw an error after logging or doing its error pipeline logic, you can have an Exit snap throw an error at the end of the error pipeline at any particular threshold of docs(0 if you want it to happen every time) and that should give you your desired behavior if I understand the ask correctly.

[jcampion]

Thanks @cjhoward18. I will investigate more about the desired behavior and apply your suggestion if that’s what’s needed. I think in this case the error pipeline itself was misconfigured and hanging. Thus, the calling pipeline was still showing running (or successful completion), but if you looked into the statistics there were errors. Once the configuration is fixed it should either succeed or fail, and the hanging should no longer be a problem. Thanks again for your quick response.

[tlikarish]

jcampion:

Thus, the calling pipeline was still showing running (or successful completion), but if you looked into the statistics there were errors.

I don’t think this is documented in the error pipeline documentation, but this is actually the expected behavior. Say you’re processing 500 documents and the 100th document in is munged and will result in an error when processing. A user may still want to execute on the other 499 valid documents and record which document is causing an issue. Additionally teams generally want to standardize on the error handling, so error pipelines allow all pipelines to process errors in a standard way.

When viewing a pipeline configured with an error pipeline, you can view the errors by looking at the pipeline execution stats. These show how many documents pass through the output and error view respectively.

As @cjhoward18 mentioned, you can add an Exit snap to the error pipeline if you want the parent pipeline to fail and stop processing. Here’s an example of that if you want to try it out.

projects_error-pipeline-example.zip (2.5 KB)

You can also modify the error handling behavior of each snap individually.

Hope this helps clarify and give you some starting points when building out the error handling in your integrations.

Best