Forum Discussion

Szymon's avatar
Szymon
New Contributor II
5 years ago

Using "Error Pipeline" - parent pipeline does not stop, how to stop it? error-pipeline called at the end?

Hi SL gurus! Pipeline got Error Pipeline defined in properties(for logging/alerting). When error occures, in e.g. Mapper, main pipeline does not stop and execute error-pipeline, but runs further....
Supratim's avatar
Supratim
Contributor III
5 years ago
Copy LinkReport

@mtran21 If you are moving one folder to another in same server can use File Operation snap. Else you can create 2 pipeline (recommending to track the processed file name and parallel processing ) -

  1. Directory Browser to poll list of File available, then pass file path details to child pipeline. Use file name in pipeline execution level, so you can track the processed filename from dash board and make no of execution count at least 10.
  2. In child pipeline, read the file (file url should comes from parent) then Write to your target folder. You should pass target folder location and filename from parent pipeline.
  3. If you want to delete from source , then use delete snap.

If you need to listen the source file path in particular frequency, then use file poll snap and make time out as -1, instead of Directory Browser. Rest should remain same.

  • tlikarish's avatar
    tlikarish
    Employee
    5 years ago
    Copy LinkReport

    I don’t think this is documented in the error pipeline documentation, but this is actually the expected behavior. Say you’re processing 500 documents and the 100th document in is munged and will result in an error when processing. A user may still want to execute on the other 499 valid documents and record which document is causing an issue. Additionally teams generally want to standardize on the error handling, so error pipelines allow all pipelines to process errors in a standard way.

    When viewing a pipeline configured with an error pipeline, you can view the errors by looking at the pipeline execution stats. These show how many documents pass through the output and error view respectively.

    As @cjhoward18 mentioned, you can add an Exit snap to the error pipeline if you want the parent pipeline to fail and stop processing. Here’s an example of that if you want to try it out.

    projects_error-pipeline-example.zip (2.5 KB)

    You can also modify the error handling behavior of each snap individually.

    Hope this helps clarify and give you some starting points when building out the error handling in your integrations.

    Best