Forum Discussion

maddog0's avatar
maddog0
New Contributor II
7 months ago
Solved

Error pipelines and pipeline execute

I am currently working on redesigning our error handling functionality.  My basic design was to write all errors to a sql server so we can have a dashboard of pipeline issues in the last 24 hours.  As a proof of concept I set this up on one of our more complex pipelines that has multiple calls to several different child pipelines in order to perform some processes in parallel.  

I know the data set I'm working with and know it has exactly 5 errors in it so when I added the error pipeline to everything I was expecting to see 5 lines, but instead I had 40.  I'm guessing this happened because I put the error pipeline on both the parent pipeline and the child pipelines.  I think the obvious solution would be to only have the error pipeline on the parent pipeline and have the child pipelines just ignore errors, but I was hoping the community could inform me of what best practice on this truly is since I can't seem to find any document that says how you should configure error pipelines when you are working with pipeline execute snaps.

Also a secondary question, if I only have the error pipeline in on the parent is there any good way for me to pass the actual name of the snap that caused the error back to the parent so it can be sent to the error pipeline as well?  currently it looks like the error documents loaded to my table only get the name of the pipeline execute snap returned, but I'd really like to get the name of the snap in the child pipeline that caused the error since I think it will make debugging much simpler.

 

1 Reply

  • JensDeveloper's avatar
    JensDeveloper
    Contributor II
    7 months ago
    Copy LinkReport

    Hi maddog0 ,

    I also wondered what the best practices are for error handling. But here is my take and experience on the error handling.
    In some use cases I am using an error pipeline in the global shared folder and using this pipeline as the error pipeline for a lot of production pipelines.
    In the production pipelines we configure some error pipeline parameters to pass to the error pipeline:

    Besides that, when a pipeline errors it also send error data to the error pipeline, like the snap name the error the payloaddata, ... 
    Below rough draft version of what to save in the database

    So I capture all of this and put this also in and SQL server database to use in an error dashboard and to use for other endpoints.

    The only downside to this logic is that the error pipeline parameters need to be configured in every pipeline to captur ruuid, pipestarttime, projectpath and pipelinename. The other data like snap name an the error is always passed when a pipeline error occurs.


    Let me know if this helps