CSV Parsing Error, More Columns Than Data

[alex_panganiban]

I am experiencing a parsing error that I would like to be able to gracefully handle before the data streams into the CSV Parser snap. We receive a file from a partner where the last row doesn't actually contain any data, but instead, it has a hidden character <0x1a>. See the attached file and screenprint. The result is that a "more columns than data" parsing error is generated.

We are removing BOM's prior to the CSV Parser. I was wondering if someone can suggest a similar technique for removing this <0x1a> character in the last row. I prefer to proactively code for this condition prior to the CSV Parser, so that in the case there are valid "more columns than data" in other files or on other rows, that they will not ignored by our exception logic.

Thanks, Alex

[bojanvelevski]

Hey @alex_panganiban,

You can trim the record to fit the headers, which will result in additional record with the hidden character as a value for the first column, in this case "employee_id".

From here, use Filter snap to exclude the records that doesn't have a "first_name" for example:

!$.hasPath('first_name')

Let me know if this works for you.

Regards,

Bojan

[koryknick]

@alex_panganiban - you can also use the "When errors occur" property in the Views tab of the CSV Parser snap to ignore errors during parsing or even route bad records to an error view and handle them more gracefully.

Hope this helps!

[alex_panganiban · Accepted solution]

Hey Y'all, thank. you so much for your responses. I made a discovery of something that I wasn't aware of before asking this question. When the CSV Parser detects an error, one of the fields that is outputted in the error view is $original.record. Now that I know about this field, I can leverage it to determine which errors need to be reported and which errors may be ignored. Initially I had wanted to handle my error condition before the CSV Parser, but now I can handle it just as effectively post CSV Parser.

Again, thanks so much for your support. I really love this facility as I know I can always count on the community to help me solve any challenge.