Does the CSV formatter snap aggregate the input documents

[walkerline117]

Hi.

If we have 5 input documents for a CSV formatter snap, how many output documents I will get from the CSV formatter?

My observation now is that CSV formatter snap will group all input documents into one and output only one big CSV file.

Is there a way to change the behavior of this? In my example, if I have 5 input documents, how to config so CSV formatter snap can output 5 output documents/files?

Thanks

[walkerline117]

yes,sorry i didn’t use clear words, but you are right, what I mean is a document with an array of values.

[walkerline117]

So in my use case, its simple: I have 5 input documents, each document is an array of values.
e.g. the first array has 3 values, the 2 array has 2 values.

after the CSV formatter, i want to see 5 documents/files( the first one should have 3 rows, the second one should have 2 rows…)

The 5 documents comes out of a mapper snap, then I tried to put a CSV formatter snap behind the mapper, i noticed that the number of output of the CSV is not 5, but 1 document contains all values across all arrays.

Then I tried to move just the CSV formatter to a subpipeline, I got the same result.

[tstack · Accepted solution]

walkerline117:

So in my use case, its simple: I have 5 input documents, each document is an array of values.
e.g. the first array has 3 values, the 2 array has 2 values.

Since the CSV Formatter currently doesn’t work with arrays, you will need to use a child pipeline that splits the array into separate docs that go into a CSV Formatter.

I’m attaching a couple of pipelines to demonstrate this flow:

FormatArrayToCSV_2019_02_28.slp (5.6 KB)
WriteCSVFiles_2019_02_28.slp (5.4 KB)

The FormatArrayToCSV pipeline is the child. It splits the “$array” property in the incoming document and then passes the results to a CSV Formatter and FileWriter. You’ll need to change the JSON Splitter to split your array property.

The WriteCSVFiles pipeline is the parent. It just contains some mock data and feeds it into the child pipeline.

[walkerline117]

This really helps.

Thanks!

[Szymon]

Hi @walkerline117

To me, this snap behaves strange, especially as you see its very different than any other formatter snap of files related snap.

Yeah, SL is not the sharpest tool in the shed. So CSV formatter collects all input documents to one.
JSON formatter has the option, not to do it. So why CSV formatter does not, I wonder 😉

Either way, to anyone, who struggles with this ’ great feature’. Maybe mapping to binary directly would be a solution for you. See screenshots below. Of course, you need to format your content first to particular format.
Well, I do not lose hope, that we get SL v1.0 before I’m retired.

cheers!

Document->Binary

make sure you map to $content: