Forum Discussion

userg's avatar
userg
New Contributor II
2 years ago

How to split files based on size(less than 20mb per file) when writing into S3

Hello, I need to write files into S3 by keeping file Size(<20mb) ,rather than giving number of records batch limit using pipeline execute. My input is dynamic sometimes 1k records write file >20mb ,...