REST POST A File?

Seems like a silly question, but I have a pipeline with a REST POST snap like below which posts a file pulled directly from the SLDB filesystem and posts it to a URL, specifying an Upload File and Upload File Key. This seems to work fine…

But my file won’t really be coming from the SLDB filesystem, so I want to pretend my file is coming in via prior snaps in the pipeline and reference that in the REST POST using the HTTP Entity. So instead I’m loading the file separately and then running it through a BASE64ENCODE, mapping that to the same Upload File Key and then trying to REST POST that, but doesn’t work.

Any ideas?

Hi Chris,

I’ll let the SnapLogic folks reply officially, but a while back I made a REST Post “Binary Form Input” custom snap that’d probably fit your use case. Happy to share and problem solve with you to get it to work for you.

Regards,
Andrew

2 Likes

Hi Andrew,

Yes please. If you have an example, I would love to check it out.

Thanks!
Chris

Hey Chris,

Want to email me at andrew_holbrook@mckinsey.com? I can’t attached the snap pack as is to this platform.

Thanks!
Andrew

Thanks. I sent you an email.

In the document conversion snap you dont need to encode. Just specify document type as conversion.

Hi Naveen,

Thanks for the suggestion. I’ve actually messed around with all the options in the Binary-to-Document snap, and it doesn’t work with any of them. I cannot specify “DOCUMENT” as the conversion type because it is a BINARY (.wav) file. I can choose NONE or ENCODE, but neither of those work with the downstream post.

On the REST POST snap, I can specify an Upload File directly (from FILE or SLDB only) and it will correctly construct and post a multi-part mime message using the file. I’m trying to achieve similar capability with a binary file already being processed in the pipeline, not pulled from FILE/SLDB at that point.

Hello,

Due to Java memory segment limitations, files being posted to an endpoint that require a multipart post must be streamed from a file.

Please note that a file upload with the REST post snap equates to a multipart post, which is not the same as trying to post a string of binary in the body which would be just a standard post, not multipart. Most API’s which are expecting files do not expect the latter and this would not work for them.

REST Post requires the file to be localish(something it can access without credentials) so that it can stream the data from the file to the endpoint. It is unsafe and unperformant to essentially build a big string of binary and try to post it somewhere, multipart or not.

If your string is too large it can (1) crash your snaplex node requiring a reboot (2) not work because part of its missing and lead to a corrupted file (3) Even if 1 or 2 isn’t a problem lead you to file-encoding blackhole and corrupted files because you’re translating in (and possible out) of UTF-8 From whatever source encoding that’s coming from.

Interesting…

So, if my pipeline already has a file, say something posted in via triggered event, or maybe created in the pipeline itself via a formatter type snap - then I would need to first write that binary file to local FILE/SLDB storage, and then reference it in the REST POST snap if I wanted to post that file? If so, the security implications are unfortunate.

If that’s required, maybe the snap itself should handle writing the binary data to a temporary file and then streaming from it within it’s own “black box” logic … Maybe there could be a new BINARY REST POST snap…

May I suggest then if you have an issue with writing documents to local or SLDB to write to a temporary file name not indicative of the contents and build into your pipeline the ability to clean that temp file afterwards. That way you can function with the REST post snap in multipart mode and not leave artifacts remaining on your filesystem whatever that may be.

When you use postman or cURL or anything really to make a multipart post you stream from a file local to the client making the connection. That’s really the only sustainable way to multipart post files that are of nontrivial sizes. The snap is no different, it needs to stream from a file, and its context of local is SLDB or on the node filesystem. Because the snap only takes one set of credentials (which are for the post target), its not possible for it to reach out unless the URI you point it as has no authentication in place.

The reason binary rest post snap is not really feasible would be you loose all of your other data in the document. When you have a binary output in SnapLogic you only have that binary stream and the headers describing that stream. You’d lose any other data fields you’d want in that document and then it couldn’t be used to configure the post snap, meaning all your setup would have to be static (which is not great )or need to be parametrized via pipeline parameters (which is also not very flexible).

if you’d like to send me an email at dwhite@snaplogic.com, I can share a script I’ve written which does multipart posting and does download to local file system, upload as multipart (both file and any form data you want), and cleanup. The caveat again here being that the callout for the download must be no auth, but maybe you could co-opt it for your own designs.

Hi,

Can you please share the script.

We have similar requirement where I am posting zip files to microsoft CRM recurring integration Azure blob storage.

We are Utilizing https://docs.microsoft.com/en-us/dynamics365/operations/dev-itpro/data-entities/recurring-integrations

I already sent you an email.

Thanks,
Piyush

@dwhite, I also need the pipline/script

@ dwhite@snaplogic.com Can you please share script / pipeline for this?