Hello @aditya.gupta41,
Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.
Then simply construct the filename in the file writer, by adding the parameter :
"TEST_" + _parameter + ".txt"
Hello,
The batches is managed by Pipeline Execute.
That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂
If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:
Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"
