Hi All,
I am trying to add dependency between the pipelines using pipeline execute snap and adding a join to wait for parallel pipelines to finish then starting the next.
issue: problem with this solution being that the pipeline after the join keeps executing again and again for each document.
Can you please help me with the solution what is the best way to add dependency between pipelines or tasks.
You can place a Tail snap after the Join to ensure that the downstream pipeline runs only once, after all the upstream pipelines have run. A Union snap can be used instead of the Join snap to make it easier to configure, the Tail snap is required in that case also
Thankyou so much for the help. It is working now.
Hi Team,
I have another question I was able to run two pipelines in parallel and add a union and tail task to achieve the dependency. My question is that after the 3 completed i want to run 2 more pipelines in parallel how can I do that.I tried below but all 4 pipelines are running together
2 before first union and 2 before next union all are running together.
If you use a copy after the tail, it will cause a data flow, the pipeline executes won’t start until they have input data so should not start until the tail has output data.
I’m not quite clear on the configuration of your pipeline and how it’s behaving. Are the child pipelines before the Join generating more than one output document? How many documents is the Join outputting?
yes the first two pipelines are uploading data to tables and then the last one just takes a snapshot. I need everything to run once not document by document.
You can consider it that I want to create the flow of tasks as tables have dependency It should not be on the documents.
I am also trying task execute but not much of success.
Sorry, I’m still not clear on how your master pipeline is working. Are the child pipelines before the Join generating more than one output document? How many documents is the Join outputting? Maybe export the pipeline or take some screenshots and attach it so we can see the details.
You should be able to build the child pipelines so that they don’t have any unlinked outputs. In which case, the PipeExec will only output one document with the status of the child execution. So, as I mention, I’m not clear on why there are multiple documents flowing out of the Join.
Hi Team,
Can you please update on the query. How can dependency be added on pipelines.
