Separate file per record

[jason_steindorf]

How would you create a separate file per record instead of the standard file with multiple records

[darshthakkar]

@jason.steindorf: You can use “Group by N” snap and use the below settings:

[koryknick]

@jason.steindorf - what output format are you looking for? JSON Formatter has an option to output each document as a binary stream.
You could also use the binary header for some formatter snaps
Otherwise, you could probably accomplish this using Pipeline Execute with the “reuse” option disabled so that each document is called by a new process

[jason_steindorf]

The output im looking for is csv.

I want to write 1 record per file.

The current pipeline is such as that all records are written to 1 file.

[darshthakkar]

@jason.steindorf: As suggested earlier, you can use a “Group by N”, and then a pipeline execute to write individual files in any format you want.

There would be a parent pipeline where you put “Group by N” as the second last snap, Pipeline execute as the last snap. Your child pipeline will have a mapper --> csv formatter --> file writer

Please try using “Group by N” and you’ll see how those records get split into individual records.