This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
FAQ
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

split target csv file into more smaller CSV's

Go to solution
SL12345
New Contributor III

a week ago

Hi Snaplogic experts, 

my pipeline is creating one big CSV file (50k rows) as the result. Collegues asked me, if it is possible to split this big CSV file into smaller ones (create csv files with thousand rows)

My question is, how to make  more csv files as an output ? 

my idea is: 
get number of all  rows for output,  using Math.floor function  get how many iterations are needed and then make a loop in order to split rows into  csv files first file - row  1 - 1000 second file 1001 - 2000 and so on .... 

 

or is there another better approach? 

Thank you

0 Kudos
Reply
1 ACCEPTED SOLUTION

Go to solution
koryknick
Employee
Employee

a week ago

@SL12345 - You can also use the Pipeline Execute snap and set the Batch Size value to your desired number of records in the target.  Create a child pipeline that is just a CSV Formatter and File Writer snap and call that child in your Pipeline Execute.  

Basically, this passes the number of records specified in the Batch Size property to an instance of the child pipeline which can create a file of those records and finish, then the parent will start a new child with the next "batch" of records, and will keep iterating until all input records are consumed.  Simple data chunking of your original file.

Hope this helps!

View solution in original post

Reply
5 REPLIES 5

Go to solution
SL12345
New Contributor III
In response to koryknick

Thursday

Hi,  thank you for your answer - thats the way how i did it - how i splitted rows into multiple files

0 Kudos
Reply