Forum Discussion

heidi_andrew's avatar
heidi_andrew
Contributor
3 years ago
Solved

Csv file with varying number of columns in each row

my csv file has several rows for each person on the file. The specific rows have different number of columns in each row. CSV file reader is having trouble with it. Is this impossible to read for thi...
  • dmiller's avatar
    dmiller
    6 years ago

    It works on a subset of data (default is 50 documents, but configurable to some degree by the admin), so if the data is changing, what displays in preview will change as well.

    As for stopping a pipeline, you should be able to edit a Snap and set Snap execution to Disabled which would stop the pipeline at that Snap (assuming a linear pipeline).

koryknick's avatar
koryknick
Employee
3 years ago

@heidi.andrew - here is another pipeline that provides a couple more examples how this could be done. In this pipeline, there are two examples - the first is similar to what @alchemiz has done, but with an added step to simplify the syntax of the record conversion into an array then object. The second path steps it down even farther using all basic snap functionality to split the input stream so it can be parsed separately based on the input row type.

I hope this helps!
Community 15513 - Multi-row CSV_2023_02_22.slp (24.3 KB)

  • alchemiz's avatar
    alchemiz
    Contributor III
    3 years ago

    Hi Sir Mohammed,

    Good day, for polling you can always use the file poller then set a trigger file in SLDB

    Attached PoC pipeline and a trigger file

    poller.txt (36 Bytes)

    canvas 001_2023_01_10.slp (8.9 KB)

    Thanks,
    EmEm