How to write a CSV using HDFS Writer

I’m attaching an example pipeline that generates a separate JSON file for each document in a particular category:

FilePerCategory_2017_11_17.slp (7.0 KB)

Here’s some detail on what’s going on in the pipeline:

• The Group By Fields Snap can be used to do the category grouping (make sure the input documents are sorted by category, otherwise I think you’ll get multiple outputs for the same category).
• Once the categories are grouped together in a single document, you can write them out to a file using the JSON Formatter. This JSON Formatter has the ‘Format each document’ option checked, which means that a separate binary output document will be generated for each input document. We want this since every input document will be the collection of items in the category and that’s what we want in each file. In addition, we set the ‘Content’ property to ‘$group’ since that contains the array of items in the category. The last thing for the formatter is to compute the file name based on the category using the ‘Binary Header Properties’ to set the ‘content-location’ property in the binary output document header.
• Finally, there’s a File Writer that writes out a file using the name as found in the ‘content-location’ property.