Unable to do error view writing as the error record is not showing in preview

[amardeep2021]

Hi All,

I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execute the pipeline, I am getting failure saying few incorrect field values or record incorrect out of the billions. I want to write them down using error view. But, as we have limitation on preview during validation, I couldn’t pass them by selection in subsequent snaps after error view.

My objective is to write down all the error records with header, reason and resolution in a file for further analysis. Is there a way to do that as my error records are after 2000 preview limitation and I could have options to select required fields before validation.?

Please help.

Regards,
Amar

[bojanvelevski]

Can you do one more, from the CSV formatter?

[amardeep2021]

Hi Boja,

PFA of csv formatter.

[amardeep2021]

Does anyone know the reason for not getting the column values in the output file? Anyone knows how to resolve this glitch?

[amardeep2021]

Any tips? Please help.

[del]

It seems possible to me that you have 30K empty rows in your input. If $a is an empty string, then your expression isNaN(parseInt($a)) == true will evaluate to true, thus passing the empty row to the output.

Try adding a Filter snap (either after each of your CSV parsers or prior to Filter1) with the following Filter expression: ((a) => a.length)(jsonPath($,"$.*").join().replace(/,/g,"")) != 0

This filter should remove any empty rows before being processed downstream.