Showing results for 
Search instead for 
Did you mean: 

Help On Creating Pipeline To Output Errors

New Contributor

Good day Community

I am building a number of pipelines to copy records from a number of tables to Azure storage and if a particular date value of “0000-00-00” is found in any date field in any table I need to output this in a separate error log file in addition to outputting the valid transformed values. The pipeline is below and the tables have at least 20 date fields and if a date of “0000-00-00” occurs then I need to create a log file with the format below and also upload this to Azure storage. I added a sequence number for each row.

So for each run of the pipeline i need to output the valid data and also details of each column that had invalid dates. What is the best way I can design my pipeline to achieve this?? Thank you!!!


1,Process CFPOL,Mapper,Date Validity,FRST_CVG_ENHC_DT,0000-00-00,07-03-2019,

Output_Error Upload_Transform_Output_Error



Attached the sample pipeline and the data file have used. Tweak the pattern in the data validator to suit your date filed value. validate_date_field_2020_09_29.slp (8.8 KB) sample.txt (404 Bytes)