07-04-2022 10:08 PM
Hello,
I am facing issue while reading a huge file (more than 2 GB) in size. I want to read the file and zip the file.I tried in 2 ways and in both i am getting the error.
Method 1: After file reader snap, I am using a Binary to Document (Encode or Decode is set to NONE and also tried with BYTE_TO_ARRAY)snap followed by a mapper and then
Document to Binary Snap(Encode or Decode is set to NONE and also tried with BYTE_TO_ARRAY) .
In mapper following expression is used to map the data :
$[‘content-location’].substr([‘content-location’].lastIndexOf(“/”) + 1) → $[‘content-location’]
$content → $content
In this case I am getting “Cannot read more than 2,147,483,647 into a byte array” error.
Method 2:
After File reader snap, in mapper i have used binary i/p and o/p view and used the following expression to map the data :
$[‘content-location’].substr([‘content-location’].lastIndexOf(“/”) + 1) → $[‘content-location’]
$content → $content
After few seconds of execution, the pipeline is getting hung and I am getting error “Lost contact with Snaplex node while the pipeline was running”
Is there something whcih I am missing ??
07-05-2022 06:04 AM
I have found a solution for this, before converting the Binary Data to Document I have used a Compress snap to compress the data (GZIP) and then Decompress the data(GZIP) before writing into zip file writer.
This is not having any issue and the data is getting written to the file.
Hope this Helps !!
If anyone has other solution please let me know…
05-30-2024 10:05 AM
Thank you for the above proposed solution. I am new to Snaplogic, and wondering on the Binary to Document snap, how the output is in binary? The current Binary Document output view is in document type and I don't see option to change the output type. The pipeline you have showcase is having the output in Binary. Was it something like that before? and it got updated to have only document output view with the latest snap of Binary to Document? 🙂
Technically, the Binary to Document is self explanatory that says it converts from Binary to Document output stream. I am also facing the same issue of exceeding the bytes size when the binary dataset flows through the Binary to Document snap, I am able to compress before passing it down to "Binary to Document" snap, but how to decompress it if the output from it is in Document format?
06-20-2024 11:52 AM
Its mapper snap and change the view tab to binary/doc as needed