Designing a pipe to prevent a commit if any record fails

[NeverTooOldToLe]

I’m processing a CSV file and writing the file to an application. This part is simple mapping. However, if one of the records fails (possibly from validations in the application, possibly the last record in the file), none of the records should be written to the application. Is that kind of error handling even possible?

[rdill]

If the application does not support a two phase commit model for transaction processing, the only way to accomplish what you want to do is to write the output to a staging area. You could do a kind of “u-turn”, meaning, read the csv file, process it in the pipeline to validate the data, writing it back to the source file system with a different name, then once you have completed the processing, then you read the processed data. This is a good use case for pipeline execute. The first pipeline processes the data, if successful it calls the child pipeline to write the data to the target.
Note, the staging area can be anywhere that makes sense. The only limitation or concern should be making sure the staging area has enough space to handle the load.
Hope this helps,
Rich

[NeverTooOldToLe]

Thanks Rich,

This seems like a reasonable approach. Without a two phase commit, a complete solution may not be feasible.

There would still be some (small) chance that the validation in the pipeline doesn’t catch all the possible errors that the application could generate. I don’t think this is a limitation of the integration platform. This problem is just a tough nut to crack.