Move data from one child to another child pipeline


Is there any way to send data file from one intermediate child pipeline to another?


Hi @abhinavjha, did you try using Pipeline execute snap in your child pipeline too in order to route the data to another pipeline?

But that another pipeline is already getting data from pipeline execute of master pipeline.

I believe you can still use either of Pipeline execute, Task Execute or ForEach in this use case.
Are you generating any files in your first child pipeline that can not be passed into a pipeline execute that you must deploy in the child pipeline?

@darshthakkar - The Task Execute and For Each snaps are deprecated and we are recommending use of the Pipeline Execute.

@abhinavjha - You could pass the filename as a pipeline parameter through the Pipeline Execute and the child can read the file contents. If that doesn’t work for you, perhaps we need a bit more clarity on what you are trying to accomplish.

1 Like

I need to accumulate error from all child pipelines to one single file. For one pipeline I can do this by using error view and creating a file. But what about errors from other child pipelines.

Also for clarity I have a master pipeline which is taking one input file and multiple child pipelines are working by that sequentially.

@abhinavjha - are you using the error view on the Pipeline Execute? If yes, you could use the Union snap to combine the error views from all children before writing to file.

1 Like

I am using error view in the child pipeline not in the pipeline execute of parent and that child is giving different output to parent ( the output I need ). I also need error file from all the child pipeline as one single document.

OK - so each child pipeline will write to the error file, right?

What are you doing with this error file? Does it need to be permanently written to any particular location, such as AWS S3, Windows share, or SFTP? Or are you doing something with it (like email it as an attachment), then delete it?

These answers will help dictate your solution.

We are trying to collect all error information as one single file from all the child pipelines and store them on a shared location. A different team will do the analysis on those error record and provide us with correct records in next cycle.

For now we are using filewriter to get error in each child pipeline. Is there any way so that I can use filewriter in a separate child pipeline which can append all the error files it receives from other child because file writer only overwrites and not append.

Master β†’ Child Pipeline 1 – Produces Error -1
β†’ Child Pipeline 2 – Produces Error - 2
We want to merge Error 1 and Error 2 in a single file.

The File Writer can append, depending on the protocol (i.e. FILE, FTP, FTPS, and SFTP - see the online documentation). If you are not using one of these protocols, I might recommend that you write the separate error files in each child to a temporary location, then have the main pipeline read and combine the error file contents for the final destination.

1 Like