cancel
Showing results for 
Search instead for 
Did you mean: 

Dependency between pipelines

sanam
New Contributor

Hi All,

I am trying to add dependency between the pipelines using pipeline execute snap and adding a join to wait for parallel pipelines to finish then starting the next.

issue: problem with this solution being that the pipeline after the join keeps executing again and again for each document.
Can you please help me with the solution what is the best way to add dependency between pipelines or tasks.

10 REPLIES 10

sanam
New Contributor

Hi Team,
Can you please update on the query. How can dependency be added on pipelines.

akidave
Employee
Employee

You can place a Tail snap after the Join to ensure that the downstream pipeline runs only once, after all the upstream pipelines have run. A Union snap can be used instead of the Join snap to make it easier to configure, the Tail snap is required in that case also

sanam
New Contributor

Thankyou so much for the help. It is working now.

sanam
New Contributor

Hi Team,

I have another question I was able to run two pipelines in parallel and add a union and tail task to achieve the dependency. My question is that after the 3 completed i want to run 2 more pipelines in parallel how can I do that.I tried below but all 4 pipelines are running togethernewflow
2 before first union and 2 before next union all are running together.

cstewart
Former Employee

If you use a copy after the tail, it will cause a data flow, the pipeline executes won’t start until they have input data so should not start until the tail has output data.