File Splitter
Team,
I have a requirement to read a file and split the file dynamically based on file size into N number files to be written on AWS S3. Example: sales.txt → sales1.txt, sales2.txt, sales3.txt etc…...
Forum Discussion
Thank you. My objective is to, read total records and see if I can split them into multiple files dynamically without re-reading the data by passing per file record count like 1000 as a variable. Need to think logic, how I can read 50K records file and get a counter of 5K to pass this counter variable to dynamic file name at run time. Thinking towards similar lines but though of checking if any easy way / setting to achieve this functionality.
Hi Srini,
I think we can split records based on group by n snap.
value of group by size should be equal to number of records per file.
attaching pipeline that might help you to achieve what you are trying to achieve.
testFileSpliter_2019_02_08 (1).slp (12.5 KB)
writeFile_2019_02_08.slp (8.1 KB)
Thank you Ajay. I will take a look but seems like it will address my issue.
-Srini
