Separate file per record

How would you create a separate file per record instead of the standard file with multiple records

@jason.steindorf: You can use “Group by N” snap and use the below settings:

@jason.steindorf - what output format are you looking for? JSON Formatter has an option to output each document as a binary stream.
You could also use the binary header for some formatter snaps
Otherwise, you could probably accomplish this using Pipeline Execute with the “reuse” option disabled so that each document is called by a new process

The output im looking for is csv.

I want to write 1 record per file.

The current pipeline is such as that all records are written to 1 file.

image

@jason.steindorf: As suggested earlier, you can use a “Group by N”, and then a pipeline execute to write individual files in any format you want.

There would be a parent pipeline where you put “Group by N” as the second last snap, Pipeline execute as the last snap. Your child pipeline will have a mapper --> csv formatter --> file writer

Please try using “Group by N” and you’ll see how those records get split into individual records.

@jason.steindorf - Thanks for the clarification.

Here is a small example pipeline that I believe will do what you are looking for without the requirement for Pipeline Execute or Group By N snaps. Basically, I’m manually formatting the data values as CSV using the Object.values() method to get the array of values, then joining those values using the Array.join() method to concatenate the values into a string. This is sent to the $content Target Path, and I’m setting the content-type and providing a filename in the binary header properties for the File Writer. The Mapper is using a “binary” output view to feed into the File Writer.

So there are several tricks in here, but it’s very low-weight and produces one output file per document.
Community 13446 - Multi CSV Output_2022_09_20.slp (5.6 KB)

1 Like

Thanks @darshthakkar , i got it working