Hi All, I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view. My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.? Please help. Regards, Amar

Hi @amardeep2021, You can always use an SLDB file for debugging. Add a JSON Formatter to the error view, and File Writer to write the file on SLDB. After the file is stored, you can use it to construct a solution for storing the errors in a different format/location (replace the file writing part) Hope this helps, Bojan

Hi Amar, Good day, you can also use the Data Validator snap where you can set specific rules and even pattern Thanks, EmEm

As the size of the file is bigger, storing in snaplogic may not be possible. As alternate, I am trying to identify non numeric values coming in numeric fields in upstream documents and try to separate to address the non numeric records for a particular field to fix this. I tried filter with isNaN($a) == ‘true’, it didn’t work. Any suggestion? PFA of snaps I used. Regards, Amar

That’s why I suggested to use the error file and construct a solution that will write the errors elsewhere. You can try and use the instanceof or typeof operators in the filter: $a instanceof Number typeof $a == ‘number’ Or modify your actual expression: isNaN(parseInt($a)) == true What you were trying so far is not working because you are checking if the $a field is NaN. NaN is a response if you’re trying to parse a value that is Not a Number - NaN. That’s why you need to try and parse the value, than if the value is string or date or whatever, than you’ll get ‘NaN’ and your expression will be true. But be careful, the string numbers will also be affected, that’s why you should probably go with the operators above.

or you can also isNaN($.get(‘a’,‘isnull’)) when isNaN(null) will return false

Unable to do error view writing as the error record is not showing in preview

22 Replies

amardeep2021
New Contributor III
5 years ago
I supposed to receive only number in this field. When I try loading to redshift, I see error below. So decided to separate the non numeric records to correct them with above pipeline. It process but not values in final file. To answer your question, I am seeing data like below after union json preview.

Redshift bulk load error view message
Error loading document 5658362 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658364 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658366 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658368 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";Error loading document 5658370 : Column age : Error code 1207 : "Invalid digit, Value ‘/’, Pos 2, Type: Integer ";
bojanvelevski
Valued Contributor
5 years ago
Well, this is a string. Should this value pass the filter or not? Because with your current expression, this will be considered as a number.
amardeep2021
New Contributor III
5 years ago
You mean to say, I need to check for $age == null? As this field getting number in string and for some it’s special character(no specific pattern). I just need to filter records which don’t carry number as strin in that case. Any suggestions?
- bojanvelevski
  Valued Contributor
  5 years ago
  In that case, as I already said in my previous replies, use the typeof operator:
  
  typeof $a == 'number'
amardeep2021
New Contributor III
5 years ago
I tried about typeof expression == ‘number’ in filter, I see some junk values in first column of final file instead of actual value.
bojanvelevski
Valued Contributor
5 years ago
Add $a != null to it. Like this:

typeof $a == 'number' && $a != null

Forum Discussion

Unable to do error view writing as the error record is not showing in preview

22 Replies

Recent Discussions

trace API and proxy calls

Pagination Logic Fails After Migrating from REST GET to HTTP Client Snap

Pipeline Execute Pool size

Concat values of a field based on value of another field

JWT Configuration for SnapLogic Public API