Unable to do error view writing as the error record is not showing in preview

[amardeep2021]

Hi All,

I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.

My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?

Please help.

Regards,
Amar

[alchemiz]

or you can also isNaN($.get(‘a’,‘isnull’))

when isNaN(null) will return false

[amardeep2021]

Thanks all. I am trying to write only the records which has non numeric value for a by using below snaps and the filter condition as isNaN(parseInt($a)) == true. Pipeline completes successfully. Dashboard status showing filter input in 90 million and output as 30k. Since I have added a comment in the mapper after filter snap, I see the file with 30k rows with that comment. But I don’t see any other fields passed to the final file except the comment. I tried pass through and also pulled fields manually in mapper after filter, still same. Can anyone know the reason and possible resolution for this? Kindly help.

Regards,
Amar.

[amardeep2021]

One additional information, column headers are writing to the final file with the comment I have added in the mapper but no values written for the columns for all 30k rows. Any idea to fix this? Please help.

Regards,
Amar.

[bojanvelevski]

Try and validate if values are getting to the filter. Additionally, it would be helpful if you can take a screen shot of the maper after the filter.

[amardeep2021]

Hi Bojan,

I could see the input output count in dashboard as attached below. Same number of rows I can see in final file with header, empty values and stat column with value as ageerror. PFA of mapper screenshot. Tried pass though as well, same result.

Regards,
Amar