Hi All,
Trying to get file’s from Rest API using Rest get snap ,while getting a large file from API more than 1 GB ,Snap throws below error.
Failure: Failed to process HTTP response, Reason: SLDB does not support data larger than 100 MB, Resolution: Please check the values of Snap properties
We had the same issue. This occurs because the preview data is written to SLDB and has a 100mb limit, which is usually fine for the 50 rows of data, but for large REST responses that contain an internal array of items within a single document that limit is easily reached. This is however only a preview time error, as a full run does not create previews, so actually running the pipeline worked fine and did not cause the error.
I have encountered this issue and my workaround was save it to s3 using file writer and then read it as a normal file.
