REST GET Snap error (SLDB does not support data larger than 100 MB)

[bharath]

Hi All,

Trying to get file’s from Rest API using Rest get snap ,while getting a large file from API more than 1 GB ,Snap throws below error.

Failure: Failed to process HTTP response, Reason: SLDB does not support data larger than 100 MB, Resolution: Please check the values of Snap properties

[CMeyers]

We had the same issue. This occurs because the preview data is written to SLDB and has a 100mb limit, which is usually fine for the 50 rows of data, but for large REST responses that contain an internal array of items within a single document that limit is easily reached. This is however only a preview time error, as a full run does not create previews, so actually running the pipeline worked fine and did not cause the error.

[jaybodra]

I have encountered this issue and my workaround was save it to s3 using file writer and then read it as a normal file.