I want to validate the incoming data which will be in CSV format. I have Target table defined for each file. Now I want to develop a common pipeline which should validate the data based on the rule for each file.
Validation Rules - Not null, length and datatype. I can extract these validation information from target database dynamically using sql queries based on the file name. How do I merge this rules with incoming data ? My goal is to develop one pipeline and even if there are changes in database for new field addition or datatype change, I shouldn’t change my pipeline. my sql queries can pull the latest fields, datatype and other req information.
Basically i’m trying to verify the data in the file before I apply them to target table. So I capture error records before I perform my DML operation