Parsing CSV file that has header/trailer records throughout

Shawn.McCormick · February 13, 2019, 2:50pm

I need to parse a CSV file that has header and trailer records throughout the file in unpredictable locations. What would be the best way to try and do this?

Unfortunately these records do not have the same number of data elements as the data records so the CSV parse pipline is failing.

<data records>
01018,2019/01/13,2019/01/13,6,1049,3243,2514,4554,52
01018,2019/01/13,2019/01/13,7,21.53,79.14,138.80,100.46,62.00
Store Trailer-PUW,01018,2019/01/15,49
Store HDR-PUW,1026,2019/01/15
01026,2019/01/13,2019/01/07,1,17,192,29,278,4
01026,2019/01/13,2019/01/07,2,1018,6153,5391,6777,141
<more data records>

Any suggestions? Thanks!

Shawn.McCormick · February 14, 2019, 2:47pm

I am new to SnapLogic. I just found that I could route the error data to the error view. It’s not ideal but I can handle it that way.

Ajay_Chawda · February 17, 2019, 9:23am

Hi Shawn.McCormick
Possible to share pipeline or screen shot of pipeline how you achieve this requirement.

Shawn.McCormick · February 18, 2019, 3:35pm

For the CSV Parser pipline the default error behavior is to “stop pipeline execution”. But if you change it to “route error data to error view” it will kick out the records/documents that don’t fit the parser formatting and it will continue to process the good data.