If the application does not support a two phase commit model for transaction processing, the only way to accomplish what you want to do is to write the output to a staging area. You could do a kind of “u-turn”, meaning, read the csv file, process it in the pipeline to validate the data, writing it back to the source file system with a different name, then once you have completed the processing, then you read the processed data. This is a good use case for pipeline execute. The first pipeline processes the data, if successful it calls the child pipeline to write the data to the target.
Note, the staging area can be anywhere that makes sense. The only limitation or concern should be making sure the staging area has enough space to handle the load.
Hope this helps,
Rich
