cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Writing file while performing batch processing

aditya_gupta41
Contributor

Hello Experts,

I have a scenario where I have build a parent-child pipeline where the source has 50 records and the agenda is to process it in batch of 10, so the pipeline will run 5 times.
The child contains a file writer which will create file in sldb location. The query is I want to write file with different file name for each batch process.

For eg.
1st batch - 10 records - Filename: TEST_1.txt
2nd batch - 10 records - Filename: TEST_2.txtโ€ฆand so on

Thanks in Advance

3 REPLIES 3

bojanvelevski
Valued Contributor

Hello @aditya.gupta41,

Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If youโ€™re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.

Then simply construct the filename in the file writer, by adding the parameter :

"TEST_" + _parameter + ".txt"

Hello,

The batches is managed by Pipeline Execute.

That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline ๐Ÿ™‚

If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:

Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"