Unable to do error view writing as the error record is not showing in preview

[amardeep2021]

Hi All,

I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.

My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?

Please help.

Regards,
Amar

[bojanvelevski]

Hi @amardeep2021,

You can always use an SLDB file for debugging. Add a JSON Formatter to the error view, and File Writer to write the file on SLDB.

After the file is stored, you can use it to construct a solution for storing the errors in a different format/location (replace the file writing part)

Hope this helps,
Bojan

[alchemiz]

Hi Amar,

Good day, you can also use the Data Validator snap where you can set specific rules and even pattern

Thanks,
EmEm

[amardeep2021]

As the size of the file is bigger, storing in snaplogic may not be possible. As alternate, I am trying to identify non numeric values coming in numeric fields in upstream documents and try to separate to address the non numeric records for a particular field to fix this. I tried filter with isNaN($a) == ‘true’, it didn’t work. Any suggestion? PFA of snaps I used.

Regards,
Amar

[bojanvelevski]

That’s why I suggested to use the error file and construct a solution that will write the errors elsewhere. You can try and use the instanceof or typeof operators in the filter:

$a instanceof Number

typeof $a == ‘number’

Or modify your actual expression:

isNaN(parseInt($a)) == true

What you were trying so far is not working because you are checking if the $a field is NaN. NaN is a response if you’re trying to parse a value that is Not a Number - NaN. That’s why you need to try and parse the value, than if the value is string or date or whatever, than you’ll get ‘NaN’ and your expression will be true. But be careful, the string numbers will also be affected, that’s why you should probably go with the operators above.