11-11-2022 02:47 AM
Hello Experts,
I have a scenario where I have build a parent-child pipeline where the source has 50 records and the agenda is to process it in batch of 10, so the pipeline will run 5 times.
The child contains a file writer which will create file in sldb location. The query is I want to write file with different file name for each batch process.
For eg.
1st batch - 10 records - Filename: TEST_1.txt
2nd batch - 10 records - Filename: TEST_2.txt…and so on
Thanks in Advance
11-11-2022 02:56 AM
Hello @aditya.gupta41,
Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.
Then simply construct the filename in the file writer, by adding the parameter :
"TEST_" + _parameter + ".txt"
11-11-2022 03:01 AM
Hello,
The batches is managed by Pipeline Execute.
11-11-2022 03:11 AM
That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂
If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:
Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"