REST POST A File?

[christwr]

Seems like a silly question, but I have a pipeline with a REST POST snap like below which posts a file pulled directly from the SLDB filesystem and posts it to a URL, specifying an Upload File and Upload File Key. This seems to work fine…

But my file won’t really be coming from the SLDB filesystem, so I want to pretend my file is coming in via prior snaps in the pipeline and reference that in the REST POST using the HTTP Entity. So instead I’m loading the file separately and then running it through a BASE64ENCODE, mapping that to the same Upload File Key and then trying to REST POST that, but doesn’t work.

Any ideas?

[christwr]

Thanks. I sent you an email.

[nganapathiraju]

In the document conversion snap you dont need to encode. Just specify document type as conversion.

[christwr]

Hi Naveen,

Thanks for the suggestion. I’ve actually messed around with all the options in the Binary-to-Document snap, and it doesn’t work with any of them. I cannot specify “DOCUMENT” as the conversion type because it is a BINARY (.wav) file. I can choose NONE or ENCODE, but neither of those work with the downstream post.

On the REST POST snap, I can specify an Upload File directly (from FILE or SLDB only) and it will correctly construct and post a multi-part mime message using the file. I’m trying to achieve similar capability with a binary file already being processed in the pipeline, not pulled from FILE/SLDB at that point.

[dwhite]

Hello,

Due to Java memory segment limitations, files being posted to an endpoint that require a multipart post must be streamed from a file.

Please note that a file upload with the REST post snap equates to a multipart post, which is not the same as trying to post a string of binary in the body which would be just a standard post, not multipart. Most API’s which are expecting files do not expect the latter and this would not work for them.

REST Post requires the file to be localish(something it can access without credentials) so that it can stream the data from the file to the endpoint. It is unsafe and unperformant to essentially build a big string of binary and try to post it somewhere, multipart or not.

If your string is too large it can (1) crash your snaplex node requiring a reboot (2) not work because part of its missing and lead to a corrupted file (3) Even if 1 or 2 isn’t a problem lead you to file-encoding blackhole and corrupted files because you’re translating in (and possible out) of UTF-8 From whatever source encoding that’s coming from.

[christwr]

Interesting…

So, if my pipeline already has a file, say something posted in via triggered event, or maybe created in the pipeline itself via a formatter type snap - then I would need to first write that binary file to local FILE/SLDB storage, and then reference it in the REST POST snap if I wanted to post that file? If so, the security implications are unfortunate.

If that’s required, maybe the snap itself should handle writing the binary data to a temporary file and then streaming from it within it’s own “black box” logic … Maybe there could be a new BINARY REST POST snap…