Unable to do error view writing as the error record is not showing in preview

[amardeep2021]

Hi All,

I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.

My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?

Please help.

Regards,
Amar

[bojanvelevski]

In that case, as I already said in my previous replies, use the typeof operator:

typeof $a == 'number'

[amardeep2021]

I tried about typeof expression == ‘number’ in filter, I see some junk values in first column of final file instead of actual value.

[bojanvelevski]

Add $a != null to it. Like this:

typeof $a == 'number' && $a != null