Unable to do error view writing as the error record is not showing in preview

[amardeep2021]

Hi All,

I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.

My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?

Please help.

Regards,
Amar

[amardeep2021]

Expression and pipeline working fine and generates file with all records and column values for opposite one isNaN(parseInt($a)) == false

But for the negative scenario it’s creating file with header and only comment column with value for 90 million records.

Does anyone know why this is happening only when I use this expression isNaN(parseInt($a)) == true the filter as I have not changed anything else?

Above sent setting screenshots are same. Kindly help me to resolve this as this is very high priority for me.

Regards,
Amar.

[bojanvelevski]

Do you by any means receive only numbers, and string version of numbers from the $a field? Ex.: 1 or “1”?

[amardeep2021]

I supposed to receive only number in this field. When I try loading to redshift, I see error below. So decided to separate the non numeric records to correct them with above pipeline. It process but not values in final file. To answer your question, I am seeing data like below after union json preview.

Redshift bulk load error view message
Error loading document 5658362 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658364 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658366 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658368 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658370 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";

[bojanvelevski]

Well, this is a string. Should this value pass the filter or not? Because with your current expression, this will be considered as a number.

[amardeep2021]

You mean to say, I need to check for $age == null? As this field getting number in string and for some it’s special character(no specific pattern). I just need to filter records which don’t carry number as strin in that case. Any suggestions?