cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Using "Error Pipeline" - parent pipeline does not stop, how to stop it? error-pipeline called at the end?

Szymon
New Contributor II

Hi SL gurus!

Pipeline got Error Pipeline defined in properties(for logging/alerting).
When error occures, in e.g. Mapper, main pipeline does not stop and execute error-pipeline, but runs further. If more errors occur, they will be sent to the error-pipeline as an array.
What is more, in Dashboard, main pipeline gets status Completed, not Failed.
Is there a possibility to use error-pipeline, so error logging is easier and stop the pipeline, when an error occurs?

thanks
Simon ๐Ÿ™‚ ๐Ÿ˜„

5 REPLIES 5

jcampion
New Contributor III

Hi @Szymon. Were you ever able to solve this? I am seeing the same behavior and would appreciate any advice. Thanks!

@jcampion @Szymon

If youโ€™d like the error pipeline to also throw an error after logging or doing its error pipeline logic, you can have an Exit snap throw an error at the end of the error pipeline at any particular threshold of docs(0 if you want it to happen every time) and that should give you your desired behavior if I understand the ask correctly.

jcampion
New Contributor III

Thanks @cjhoward18. I will investigate more about the desired behavior and apply your suggestion if thatโ€™s whatโ€™s needed. I think in this case the error pipeline itself was misconfigured and hanging. Thus, the calling pipeline was still showing running (or successful completion), but if you looked into the statistics there were errors. Once the configuration is fixed it should either succeed or fail, and the hanging should no longer be a problem. Thanks again for your quick response.

I donโ€™t think this is documented in the error pipeline documentation, but this is actually the expected behavior. Say youโ€™re processing 500 documents and the 100th document in is munged and will result in an error when processing. A user may still want to execute on the other 499 valid documents and record which document is causing an issue. Additionally teams generally want to standardize on the error handling, so error pipelines allow all pipelines to process errors in a standard way.

When viewing a pipeline configured with an error pipeline, you can view the errors by looking at the pipeline execution stats. These show how many documents pass through the output and error view respectively.

image

As @cjhoward18 mentioned, you can add an Exit snap to the error pipeline if you want the parent pipeline to fail and stop processing. Hereโ€™s an example of that if you want to try it out.

projects_error-pipeline-example.zip (2.5 KB)

You can also modify the error handling behavior of each snap individually.

image

Hope this helps clarify and give you some starting points when building out the error handling in your integrations.

Best