Run error pipeline on different snaplex

Hi,

how can I run the actual pipeline on one snaplex and but error pipeline configured for it on another snaplex?

Hi,
Can you please try using a child pipeline and use pipeline execute snap in error pipeline to call the child by mentioning the snaplex detail.

we were having similar configuration earlier but as with 4.20, 8MB input limit imposed on pipeline execute snap is not allowing to do so. hence we switching to error pipelines

how above case can be archived considering this limitation ?

Hi @vaidyarm,

You could try and create a triggered task for the pipeline you want to execute on a different snaplex from the first one. In the Task’s settings you can specify the snaplex on which you want the task to execute.

Then, you may call the triggered task by using a REST Post Snap. I did a little test, and successfully passed 9,53 MB of data to the pipeline in the triggered task, by using a REST Post Snap.

Let me know if I can help you further. If you want I can share some pipelines that can illustrate this solution.

BR,
Dimitri

1 Like

Thanks @dimitri.hristovski for this design proposal, I will try this option.

Could the REST Post done can be tied to same execution ? (like we see child pipeline tied to its parent visible from dashboard). i am looking at this from auditing perspective, as in case when we search for any particular transaction status if any error occurred, it should pop up on searching dashboard.

You’re welcome @vaidyarm,

Unfortunately you can’t set the execution label for the triggered pipeline in the REST Post Snap the way that you can in the Pipeline Execute Snap.
However, in the Pipeline Execution History in the Pipeline tab in the Dashboard; the calling and the called pipeline should show up over or under each other (but not nested, as is the case when the Pipeline Execute Snap is used). So you’d have to use a meaningful name for the called pipeline by which you can search.
You then can reference the Pipeline Execution Statistics from the Status column for the appropriate item in the list.

Hope this helps.

BR,
Dimitri

1 Like

Thanks for confirming the doubt.

was thinking to have another pipeline execute snap in parallel with REST POST but not with entire payload (which is > 8MB) but just few auditing fields and call some dummy pipeline, so that in single execution we can have track of errors as well as actual errors also pushed through REST POST.

Alternatively, you could put together an error pipeline, and assign it to both the parent pipeline and the pipeline called through the REST Post. In the error pipeline you could write the information for the errors (such as pipeline label, timestamp, label of the failing snap, error, reason, stack trace etc.) to a file, to a database or send it via email.

That way you can have a single point of error handling for all of the pipelines.

1 Like

that could be done, but since we have centralized error pipeline to handle errors for hundreds of our pipelines in generic way, would prefer to adhere to those architecture standards and make changes required for this specific pipeline.
we will consider your database approach for any future requirements, thanks again !

The limit is only for the pipeline parameters. If the data is large, you can pass it as a regular document or a binary document to the unlinked input view of the child.