Forum Discussion

amardeep2021's avatar
amardeep2021
New Contributor III
4 years ago

Unable to do error view writing as the error record is not showing in preview

Hi All, I am trying to process millions of records even sometimes in billions. While validation, I am not seeing preview coming up for error view as we have 2000 limitation in preview. When I execu...
amardeep2021's avatar
amardeep2021
New Contributor III
4 years ago

Any tips? Please help.

  • del's avatar
    del
    Contributor III
    4 years ago

    It seems possible to me that you have 30K empty rows in your input. If $a is an empty string, then your expression isNaN(parseInt($a)) == true will evaluate to true, thus passing the empty row to the output.

    Try adding a Filter snap (either after each of your CSV parsers or prior to Filter1) with the following Filter expression: ((a) => a.length)(jsonPath($,"$.*").join().replace(/,/g,"")) != 0

    This filter should remove any empty rows before being processed downstream.