Process 2 Gb file
I have a very big file (around 2gb) . I have to fetch that from FTP and have to split the file into chunks (let’s say 2 million records ) and create new files in the FTP . Please suggest any effectiv...
Forum Discussion
Hi @aleung ,
We have implemented this with the sequence generator and Router. It took 1 hour to process the whole file and creates new files with 2.5 million each which is pretty good .Still I am curious to know whether we can tune this to less time .could you help on this .
This type of job is best at the endpoints instead of snap. Python is really good at this kind of job, so if you can do the splitting either prior or post transfer is best. I know this isn’t ideal, but can’t think of a better way.
