Forum Discussion

aditya_gupta41's avatar
aditya_gupta41
Contributor
3 years ago

Writing file while performing batch processing

Hello Experts,

I have a scenario where I have build a parent-child pipeline where the source has 50 records and the agenda is to process it in batch of 10, so the pipeline will run 5 times.
The child contains a file writer which will create file in sldb location. The query is I want to write file with different file name for each batch process.

For eg.
1st batch - 10 records - Filename: TEST_1.txt
2nd batch - 10 records - Filename: TEST_2.txt…and so on

Thanks in Advance

3 Replies

  • bojanvelevski's avatar
    bojanvelevski
    Valued Contributor
    3 years ago

    Hello @aditya.gupta41,

    Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
    If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.

    Then simply construct the filename in the file writer, by adding the parameter :

    "TEST_" + _parameter + ".txt"

      • bojanvelevski's avatar
        bojanvelevski
        Valued Contributor
        3 years ago

        That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂

        If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:

        Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"