This works. Thanks for your help @j.angelevski
Thank you @koryknick, the solution worked.
I happen to tweak the expression in the mapper a bit so that no extra spaces were observed:
Yes, the text file will always have “x” fields, it wouldn’t even go to x-1 or x+1.
Moreover, as there is no max value in a group size as per documentation, I can pretty much keep it to 100 for instance to handle the additional rows coming in the future. What are your thoughts?
Intentionally, kept 100 and not 99999999 as I’m not anticipating anything more than 8.
Thank you.
Regards,
Darsh
Hello @aditya.gupta41,
Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.
Then simply construct the filename in the file writer, by adding the parameter :
"TEST_" + _parameter + ".txt"
Hello,
The batches is managed by Pipeline Execute.
That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂
If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:
Example expression in the file writer: "TEST_" + lib.config.sequence + 1 + ".txt"
