Unable to do error view writing as the error record is not showing in preview
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-09-2021 06:10 AM
Hi All,
I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.
My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?
Please help.
Regards,
Amar

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-10-2021 09:13 AM
Can you do one more, from the CSV formatter?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-10-2021 10:11 AM
Hi Boja,
PFA of csv formatter.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-10-2021 01:54 PM
Does anyone know the reason for not getting the column values in the output file? Anyone knows how to resolve this glitch?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-11-2021 12:29 PM
Any tips? Please help.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎11-11-2021 01:08 PM
It seems possible to me that you have 30K empty rows in your input. If $a is an empty string, then your expression isNaN(parseInt($a)) == true
will evaluate to true, thus passing the empty row to the output.
Try adding a Filter snap (either after each of your CSV parsers or prior to Filter1) with the following Filter expression: ((a) => a.length)(jsonPath($,"$.*").join().replace(/,/g,"")) != 0
This filter should remove any empty rows before being processed downstream.
