Create Excel files larger than 100MB

[rustin]

I need to create an Excel file which is larger than 100MB in file writer. I am receiving an error message in File Writer: 

Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB

I am copying the whole message below. Is there a way to workaround this?

Failed to write to gdrive.xlsx

Resolution:

Address the reported issue.

Reason:

Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB

Hide Details...

File Writer[570b47e3a415f5440eb1e2dc_43c3cbfb-6c35-4be5-98d4-9dbe3c917864 -- c6bcdbd4-c8aa-4b4f-bb4f-19441389290f]
com.snaplogic.snap.api.SnapDataException: Failed to write to gdrive.xlsx at com.snaplogic.snaps.binary.AbstractWriter.throwExceptionCantWrite(AbstractWriter.java:739) at com.snaplogic.snaps.binary.AbstractWriter.writeData(AbstractWriter.java:595) at com.snaplogic.snaps.binary.AbstractWriter.process(AbstractWriter.java:379) at com.snaplogic.snaps.binary.AbstractWriter.doWork(AbstractWriter.java:329) at com.snaplogic.snap.api.SimpleBinarySnap.execute(SimpleBinarySnap.java:57) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:812) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:586) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:877) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.snaplogic.api.ExecutionException: SLDB does not support data larger than 100 MB at com.snaplogic.common.url.protocol.sldb.SldbOutputStream.withinLimits(SldbOutputStream.java:230) at com.snaplogic.common.url.protocol.sldb.SldbOutputStream.write(SldbOutputStream.java:120) at java.base/java.io.BufferedOutputStream.write(Unknown Source) at com.snaplogic.snaps.binary.AbstractWriter.copy(AbstractWriter.java:522) at com.snaplogic.snaps.binary.AbstractWriter.writeData(AbstractWriter.java:582) ... 14 more Reason: Possible reasons can be failure in URL connection or file access denial, detail: SLDB does not support data larger than 100 MB Resolution: Address the reported issue. Error Fingerprint[0] = efp:com.snaplogic.snaps.binary.BcA2Uzfu Error Fingerprint[1] = efp:com.snaplogic.common.url.protocol.sldb.WQwFN6-j

[rustin]

@koryknick , I am not sure how to do that. I assume I need to talk to IT whether we have some location on SFTP or S3 which can be used for that, correct? How can I then point to that location? Is it a part of the file name? My settings as of now attached. 

[koryknick]

The filename in your File Writer is a static string; click on the equal sign to enable it as an expression (i.e. it will appear darkened).  This will allow the file to be written to a temporary directory (since you are using pipe.tmpDir) on the snapex execution node and prove that the Excel Formatter is able to handle the data volume.  Please review the File Writer snap documentation to gain an understanding on how you could write to an alternate endpoint that you can safely store the file over 100MB.

Hope this helps!

[rustin]

I changed the equal sign and now it run with no errors. However, the file was not created in the GDrive. The issue is probably in File Writer. There is one file with size over 200MB on the input, but 1 file with 0 bytes on the output. Any thought on this?