Create Excel files larger than 100MB

[rustin]

I need to create an Excel file which is larger than 100MB in file writer. I am receiving an error message in File Writer: 

Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB

I am copying the whole message below. Is there a way to workaround this?

Failed to write to gdrive.xlsx

Resolution:

Address the reported issue.

Reason:

Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB

Hide Details...

File Writer[570b47e3a415f5440eb1e2dc_43c3cbfb-6c35-4be5-98d4-9dbe3c917864 -- c6bcdbd4-c8aa-4b4f-bb4f-19441389290f]
com.snaplogic.snap.api.SnapDataException: Failed to write to gdrive.xlsx at com.snaplogic.snaps.binary.AbstractWriter.throwExceptionCantWrite(AbstractWriter.java:739) at com.snaplogic.snaps.binary.AbstractWriter.writeData(AbstractWriter.java:595) at com.snaplogic.snaps.binary.AbstractWriter.process(AbstractWriter.java:379) at com.snaplogic.snaps.binary.AbstractWriter.doWork(AbstractWriter.java:329) at com.snaplogic.snap.api.SimpleBinarySnap.execute(SimpleBinarySnap.java:57) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:812) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:586) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:877) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.snaplogic.api.ExecutionException: SLDB does not support data larger than 100 MB at com.snaplogic.common.url.protocol.sldb.SldbOutputStream.withinLimits(SldbOutputStream.java:230) at com.snaplogic.common.url.protocol.sldb.SldbOutputStream.write(SldbOutputStream.java:120) at java.base/java.io.BufferedOutputStream.write(Unknown Source) at com.snaplogic.snaps.binary.AbstractWriter.copy(AbstractWriter.java:522) at com.snaplogic.snaps.binary.AbstractWriter.writeData(AbstractWriter.java:582) ... 14 more Reason: Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB Resolution: Address the reported issue. Error Fingerprint[0] = efp:com.snaplogic.snaps.binary.BcA2Uzfu Error Fingerprint[1] = efp:com.snaplogic.common.url.protocol.sldb.WQwFN6-j

[rustin]

I also tried this, with same error:

I tried running on both Groundplex and Cloud.

[koryknick]

@rustin - the SLDB (file space in your SnapLogic project folder) has a limit of 100MB.  The path specified by pipe.tmpDir is limited by the amount of temp space allocated to SnapLogic on the execution node, so you would need to check with your server admins in your environment.

However, Excel itself has a maximum record count of 1,048,576 rows and 16,384 columns per sheet.  If you need more than that, would CSV be sufficient?  The file size would be significantly more as a CSV since Excel compresses the data, but it may be worth trying.

Also note that pipe.tmpDir is transient and exists only during pipeline execution, so as soon as the pipeline completes, everything in that directory is purged.

Hope this helps!

[rustin]

@koryknick , thanks for the reply. I will check with the admins about the temp space. 

I do not need more data than what fits into Excel (I am uploading the data to Excel manually now and want to automate it), so no need to use CSV.

[rustin]

I checked with the admin and the Temp space allocate min 4 GB to Max 8 GB. That should be sufficient for the task that I am trying to automate, right? My excel files are 100MB to 900MB.