Dependency between pipelines

[sanam]

Hi All,

I am trying to add dependency between the pipelines using pipeline execute snap and adding a join to wait for parallel pipelines to finish then starting the next.

issue: problem with this solution being that the pipeline after the join keeps executing again and again for each document.
Can you please help me with the solution what is the best way to add dependency between pipelines or tasks.

[tstack]

sanam:

I am trying to add dependency between the pipelines using pipeline execute snap and adding a join to wait for parallel pipelines to finish then starting the next.

issue: problem with this solution being that the pipeline after the join keeps executing again and again for each document.
Can you please help me with the solution what is the best way to add dependency between pipelines or tasks.

I’m not quite clear on the configuration of your pipeline and how it’s behaving. Are the child pipelines before the Join generating more than one output document? How many documents is the Join outputting?

[sanam]

yes the first two pipelines are uploading data to tables and then the last one just takes a snapshot. I need everything to run once not document by document.
You can consider it that I want to create the flow of tasks as tables have dependency It should not be on the documents.
I am also trying task execute but not much of success.

[tstack]

sanam:

yes the first two pipelines are uploading data to tables and then the last one just takes a snapshot. I need everything to run once not document by document.

Sorry, I’m still not clear on how your master pipeline is working. Are the child pipelines before the Join generating more than one output document? How many documents is the Join outputting? Maybe export the pipeline or take some screenshots and attach it so we can see the details.

You should be able to build the child pipelines so that they don’t have any unlinked outputs. In which case, the PipeExec will only output one document with the status of the child execution. So, as I mention, I’m not clear on why there are multiple documents flowing out of the Join.

[sanam]

Hi ,
Below are attached screenshot of my main pipline

Each pipeline is inserting data to tables and the count can vary.
Screenshot for child pipeline

below are the join configurations

Can you please suggest is pipeline execute the best way to achieve it and how to make sure that the pipeline runs only once after the join.

Regards,
Sanam