Writing file while performing batch processing

[aditya_gupta41]

Hello Experts,

I have a scenario where I have build a parent-child pipeline where the source has 50 records and the agenda is to process it in batch of 10, so the pipeline will run 5 times.
The child contains a file writer which will create file in sldb location. The query is I want to write file with different file name for each batch process.

For eg.
1st batch - 10 records - Filename: TEST_1.txt
2nd batch - 10 records - Filename: TEST_2.txt…and so on

Thanks in Advance

[bojanvelevski]

Hello @aditya.gupta41,

Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.

Then simply construct the filename in the file writer, by adding the parameter :

"TEST_" + _parameter + ".txt"

[aditya_gupta41]

Hello,

The batches is managed by Pipeline Execute.

[bojanvelevski]

That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂

If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:

Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"