How to Report Execution Progress in Sequential Orchestration Pipelines?

Joël B. · 2026-02-23T09:51:18.346Z

Hi Everyone, "orchestration" pipeline design question here: how to report execution progress? I mean, when the project consist in sequential, exclusive steps, like 1. prepare database, 2. extract data, 3. build content, 4. publish file with content. --- For some years now, I've designed projects requiring such reporting as follow: • a top-level pipeline call sub-pipelines in sequence, with progress message in-between, and error view catching exceptions. This relies on the assumption an exclusive feedback from sub-pipelines, either one document or an error. • sub-pipelines dedicated to each step, and supposed to return only one document to the top-pipeline, using either Join / Union or Gate snap. Here's an illustration of this sub-pipeline.

Welcome to the new Integration Nation!

How to Report Execution Progress in Sequential Orchestration Pipelines? | Integration Nation

8 comments

· Sorted by Oldest

Joël B.
·
·
Here's the top-pipeline illustation:
Joël B.
·
·
And here's an illustation of sub-pipeline 1.1, ending with a Gate so that only one document goes to pipeline output
Joël B.
·
·
And now, why I'm reporting this case: as exchanged with Olivier C., recently I observed that an error on a snap will stop the pipeline, but sometimes a downstream Gate snap will still output a document, and the Pipeline Execute snap will therefore have both an error and a normal output document. This is obviously breaking the expectation of having an exclusivity between error and document, and in this case, the top-pipeline will then start to follow both paths, resulting in normal treatment to go on while reporting the error, and in fine an incomplete document was produced.
Joël B.
·
·
So, do someone else here use Gate snaps in this fashion, and do you observe the same behavior? Otherwise, do you have a similar orchestration need, but use a different design to handle sequence and progress reporting?
Spiro T.
·
·
Hey Joël B., just a simple check 😄 ... on the Gate Snap, the "Ignore Empty Stream" checkbox is checked or unchecked? If it is unchecked, the Snap writes an empty document to the output view even there is no input data going to the snap.
Joël B.
·
·
Hi Spiro T., indeed good question, and well it depends on use cases. Most of the times it's unchecked, so that during validation, the flow continues over pipelines containing snaps that write during execution only. Sometimes it's also unchecked, so that it's possible to count how many documents math a given filter, starting from 0. But my point here is during error: all my snaps are configured for "On error: stop pipeline execution", and indeed in case of error the pipeline is stopped with an error. But on such a snap failure, a downstream Gate will sometimes still output a document, even if marked as stopped in the monitor, as if the pipeline is not always able to stop the included Gate to complete execution, even if one of its inputs is failed. Said otherwise, to me it seems that the included Gate snap cannot be reliably stopped by the pipeline in case of error, which is highly undesirable.
Joël B.
·
·
This behaviour is kinda new, I first observed it beginning of December 2024 2025.
Lee D.
·
·
Hey Joël B. I was about to mention the tick box in gate snap, but Spiro beat me to it! Could the scenario you're facing be cause by one of your snaps being successful and sending to the gate first, then one of the others failing? I wonder if this makes the gate release its info back to the parent. I haven't observed anything similar myself.
👍1

Joël B.
·
·
Here's the top-pipeline illustation:
Joël B.
·
·
And here's an illustation of sub-pipeline 1.1, ending with a Gate so that only one document goes to pipeline output
Joël B.
·
·
And now, why I'm reporting this case: as exchanged with Olivier C., recently I observed that an error on a snap will stop the pipeline, but sometimes a downstream Gate snap will still output a document, and the Pipeline Execute snap will therefore have both an error and a normal output document. This is obviously breaking the expectation of having an exclusivity between error and document, and in this case, the top-pipeline will then start to follow both paths, resulting in normal treatment to go on while reporting the error, and in fine an incomplete document was produced.
Joël B.
·
·
So, do someone else here use Gate snaps in this fashion, and do you observe the same behavior? Otherwise, do you have a similar orchestration need, but use a different design to handle sequence and progress reporting?
Spiro T.
·
·
Hey Joël B., just a simple check 😄 ... on the Gate Snap, the "Ignore Empty Stream" checkbox is checked or unchecked? If it is unchecked, the Snap writes an empty document to the output view even there is no input data going to the snap.
Joël B.
·
·
Hi Spiro T., indeed good question, and well it depends on use cases. Most of the times it's unchecked, so that during validation, the flow continues over pipelines containing snaps that write during execution only. Sometimes it's also unchecked, so that it's possible to count how many documents math a given filter, starting from 0. But my point here is during error: all my snaps are configured for "On error: stop pipeline execution", and indeed in case of error the pipeline is stopped with an error. But on such a snap failure, a downstream Gate will sometimes still output a document, even if marked as stopped in the monitor, as if the pipeline is not always able to stop the included Gate to complete execution, even if one of its inputs is failed. Said otherwise, to me it seems that the included Gate snap cannot be reliably stopped by the pipeline in case of error, which is highly undesirable.
Joël B.
·
·
This behaviour is kinda new, I first observed it beginning of December 2024 2025.
Lee D.
·
·
Hey Joël B. I was about to mention the tick box in gate snap, but Spiro beat me to it! Could the scenario you're facing be cause by one of your snaps being successful and sending to the gate first, then one of the others failing? I wonder if this makes the gate release its info back to the parent. I haven't observed anything similar myself.
👍1