Cannot read more than 2,147,483,647 into a byte array

[rnarayan]

Hello,

I am facing issue while reading a huge file (more than 2 GB) in size. I want to read the file and zip the file.I tried in 2 ways and in both i am getting the error.

Method 1: After file reader snap, I am using a Binary to Document (Encode or Decode is set to NONE and also tried with BYTE_TO_ARRAY)snap followed by a mapper and then
Document to Binary Snap(Encode or Decode is set to NONE and also tried with BYTE_TO_ARRAY) .

In mapper following expression is used to map the data :

$[‘content-location’].substr([‘content-location’].lastIndexOf(“/”) + 1) → $[‘content-location’]
$content → $content

In this case I am getting “Cannot read more than 2,147,483,647 into a byte array” error.

Method 2:

After File reader snap, in mapper i have used binary i/p and o/p view and used the following expression to map the data :

$[‘content-location’].substr([‘content-location’].lastIndexOf(“/”) + 1) → $[‘content-location’]
$content → $content

After few seconds of execution, the pipeline is getting hung and I am getting error “Lost contact with Snaplex node while the pipeline was running”

Is there something whcih I am missing ??

[rnarayan]

rnarayan:

$[‘content-location’].substr([‘content-location’].lastIndexOf(“/”) + 1) → $[‘content-location’]
$content → $content

I have found a solution for this, before converting the Binary Data to Document I have used a Compress snap to compress the data (GZIP) and then Decompress the data(GZIP) before writing into zip file writer.

This is not having any issue and the data is getting written to the file.

Hope this Helps !!

If anyone has other solution please let me know…

[Himura]

Thank you for the above proposed solution. I am new to Snaplogic, and wondering on the Binary to Document snap, how the output is in binary? The current Binary Document output view is in document type and I don't see option to change the output type. The pipeline you have showcase is having the output in Binary. Was it something like that before? and it got updated to have only document output view with the latest snap of Binary to Document? 🙂

Technically, the Binary to Document is self explanatory that says it converts from Binary to Document output stream. I am also facing the same issue of exceeding the bytes size when the binary dataset flows through the Binary to Document snap, I am able to compress before passing it down to "Binary to Document" snap, but how to decompress it if the output from it is in Document format?

[diptipatel_fm]

Its mapper snap and change the view tab to binary/doc as needed