Forum Discussion

SL12345's avatar
SL12345
New Contributor III
6 months ago
Solved

split target csv file into more smaller CSV's

Hi Snaplogic experts,  my pipeline is creating one big CSV file (50k rows) as the result. Collegues asked me, if it is possible to split this big CSV file into smaller ones (create csv files with th...
  • aditya_gupta41's avatar
    aditya_gupta41
    2 years ago

    Solution:

    I first used group by snap to group userID fields. Next I put a mapper and added a custom encrypted value with expression

    'XXXX'+sl.range(4).map(x => "0123456&#X"[Math.round(Math.random()*9)]).join('')

     Then I added a JSON splitter to split the group and included value of custom encryption.

  • koryknick's avatar
    koryknick
    Employee
    6 months ago
    Copy LinkReport

    alchemiz - one warning I have with this approach is that very large files would be retained in memory through each snap.  Since the point is to split a very large file, you may find your snaplex node resources drained while trying to run this splitter.