Writing file while performing batch processing

Question

Hello Experts,
I have a scenario where I have build a parent-child pipeline where the source has 50 records and the agenda is to process it in batch of 10, so the pipeline will run 5 times.
The child contains a file writer which will create file in sldb location. The query is I want to write file with different file name for each batch process.
For eg.
1st batch - 10 records - Filename: TEST_1.txt
2nd batch - 10 records - Filename: TEST_2.txt…and so on
Thanks in Advance

bojanvelevski · Answer

That complicates things a little bit. You can keep track of processed files in a file, so the next execution will know what number to use. But that will significantly slow down your pipeline 🙂
If there are no concurrent executions of the pipeline, you can simply write down a file with the sequence number used, and set that file as a library expression on the child pipeline itself, so you could easily set the next value:
Example expression in the file writer:  "TEST_" + lib.config.sequence + 1 + ".txt"

bojanvelevski · Answer

Hello @aditya.gupta41,
Are those batches managed by the Pipeline Execute snap, or with a Group By snap ?
If you’re already using a Group By snap, you can simply add a parameter in the Pipeline Execute with a snap.in.totalCount function. That function will count the incoming batches and pass that value on the underlying pipeline.
Then simply construct the filename in the file writer, by adding the parameter :
"TEST_" + _parameter + ".txt"

aditya_gupta41 · Answer

Hello,
The batches is managed by Pipeline Execute.

Forum Discussion

Writing file while performing batch processing

Recent Discussions

Pagination and nextCursor in header

How to get filename from file reader

Generate expression file from database query

Can we generate XML file in pretty print format using native snapLogic snaps?

401 error with HTTP Client and NTLM