cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Separate file per record

jason_steindorf
New Contributor III

How would you create a separate file per record instead of the standard file with multiple records

6 REPLIES 6

darshthakkar
Valued Contributor

@jason.steindorf: You can use โ€œGroup by Nโ€ snap and use the below settings:

image

koryknick
Employee
Employee

@jason.steindorf - what output format are you looking for? JSON Formatter has an option to output each document as a binary stream.
You could also use the binary header for some formatter snaps
Otherwise, you could probably accomplish this using Pipeline Execute with the โ€œreuseโ€ option disabled so that each document is called by a new process

jason_steindorf
New Contributor III

The output im looking for is csv.

I want to write 1 record per file.

The current pipeline is such as that all records are written to 1 file.

image

@jason.steindorf: As suggested earlier, you can use a โ€œGroup by Nโ€, and then a pipeline execute to write individual files in any format you want.

There would be a parent pipeline where you put โ€œGroup by Nโ€ as the second last snap, Pipeline execute as the last snap. Your child pipeline will have a mapper --> csv formatter --> file writer

Please try using โ€œGroup by Nโ€ and youโ€™ll see how those records get split into individual records.