Move data from one child to another child pipeline

[abhinavjha]

Hi,

Is there any way to send data file from one intermediate child pipeline to another?

Thanks

[abhinavjha]

I need to accumulate error from all child pipelines to one single file. For one pipeline I can do this by using error view and creating a file. But what about errors from other child pipelines.

Also for clarity I have a master pipeline which is taking one input file and multiple child pipelines are working by that sequentially.

[koryknick]

@abhinavjha - are you using the error view on the Pipeline Execute? If yes, you could use the Union snap to combine the error views from all children before writing to file.

[abhinavjha]

I am using error view in the child pipeline not in the pipeline execute of parent and that child is giving different output to parent ( the output I need ). I also need error file from all the child pipeline as one single document.

[koryknick]

OK - so each child pipeline will write to the error file, right?

What are you doing with this error file? Does it need to be permanently written to any particular location, such as AWS S3, Windows share, or SFTP? Or are you doing something with it (like email it as an attachment), then delete it?

These answers will help dictate your solution.

[abhinavjha]

We are trying to collect all error information as one single file from all the child pipelines and store them on a shared location. A different team will do the analysis on those error record and provide us with correct records in next cycle.

For now we are using filewriter to get error in each child pipeline. Is there any way so that I can use filewriter in a separate child pipeline which can append all the error files it receives from other child because file writer only overwrites and not append.

Master → Child Pipeline 1 – Produces Error -1
→ Child Pipeline 2 – Produces Error - 2
We want to merge Error 1 and Error 2 in a single file.