Move data from one child to another child pipeline
Hi,
Is there any way to send data file from one intermediate child pipeline to another?
Thanks
Forum Discussion
OK - so each child pipeline will write to the error file, right?
What are you doing with this error file? Does it need to be permanently written to any particular location, such as AWS S3, Windows share, or SFTP? Or are you doing something with it (like email it as an attachment), then delete it?
These answers will help dictate your solution.
We are trying to collect all error information as one single file from all the child pipelines and store them on a shared location. A different team will do the analysis on those error record and provide us with correct records in next cycle.
For now we are using filewriter to get error in each child pipeline. Is there any way so that I can use filewriter in a separate child pipeline which can append all the error files it receives from other child because file writer only overwrites and not append.
Master → Child Pipeline 1 – Produces Error -1
→ Child Pipeline 2 – Produces Error - 2
We want to merge Error 1 and Error 2 in a single file.
