Good day Community
I am building a number of pipelines to copy records from a number of tables to Azure storage and if a particular date value of “0000-00-00” is found in any date field in any table I need to output this in a separate error log file in addition to outputting the valid transformed values. The pipeline is below and the tables have at least 20 date fields and if a date of “0000-00-00” occurs then I need to create a log file with the format below and also upload this to Azure storage. I added a sequence number for each row.
So for each run of the pipeline i need to output the valid data and also details of each column that had invalid dates. What is the best way I can design my pipeline to achieve this?? Thank you!!!
RowSequence,Pipelinename,PipelineTask,ErrorType,SourceField,SourceValue,DateTime
1,Process CFPOL,Mapper,Date Validity,FRST_CVG_ENHC_DT,0000-00-00,07-03-2019,
