05-12-2021 11:05 AM
I’m trying to use an API to post data to an application.
There is no account. Instead I have a specific account.
The curl API looks like this:
curl https://communicate.modolabs.net/api/v1/directories/revisions --header “Authorization: Token myToken” --form “name=Modo Test” --form “upload=@directory.xml” where “myToken” would be the token I was given.
How do I convert this to the REST Snap?
08-11-2021 11:29 AM
@mike.korcynski
Not sure on this, but you’re probably right. I will investigate! I likely won’t be able to look into this today, but tomorrow (Thu, 8/12) for sure.
08-13-2021 03:30 PM
@mike.korcynski @Sowmya_Rayavarapu @robert_parks
Here is a solution to POST-ing a file with multiple form fields.
Our REST POST snap is a swiss-army knife, and I am less familiar with all of its options (multipart related, for example). I need to better understand the “entity” plumbing too. That said, let’s evaluate something that might work – maybe not an awesome solution, but hopefully something that does work.
We want to POST more than one thing, so let’s ignore the “single file upload” fields (4 of them). We’ll leave all the entity fields empty too.
The conceptual model is that we need to create a file for every field we want to submit. If we want to upload a file and also specify two form fields, we’ll create 3 files! Files can be referenced in SLDB, or the filesystem which may be less convenient for you depending on your Snaplex filesystem access.
Here’s the top half of the REST POST snap showing the fields I left empty.
I’ve attached two simple pipelines each with a single REST POST snap.
For the first pipeline we want to upload a file (directory.xml) and also specify two form fields. Here’s the curl version:
curl -X POST http://127.0.0.1:8001/test \
--form "upload=@directory.xml" \
--form "field1=field1-value" \
--form "field2=field2-value"
We can define the form fields as files using the form field name as the “Upload-file key”. We then specify a file which contains the form field’s value. field1.txt stores the string “field1-value”. We also specify the content type as text/plain.
Here’s how the POST is received in my test server.
> POST /test
> REQUEST HEADERS:
Accept=[*/*]
Connection=[Keep-Alive]
Host=[127.0.0.1:8001]
Transfer-encoding=[chunked]
Content-type=[multipart/form-data; boundary=PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2]
> REQUEST BODY:
--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="upload"; filename="directory.xml"
Content-Type: application/xml; charset=UTF-8
Content-Transfer-Encoding: binary
<directory name="foo">
<file>file1.txt</file>
<file>file2.json</file>
</directory>
--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="field1"; filename="x"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary
field1-value
--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="field2"; filename="x"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary
field2-value
--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2--
Ok, for the second pipeline we get a bit more ambitious and double it, that is, upload 2 files, and specify 4 form fields. The second file in this case is an image (a SnapLogic logo, of course!). We also specify some of the form field files on the filesystem (ie, file:///tmp). The 4th form field is defined as json. That’s a nice part of multipart upload, each part defines its content type.
I’ve attached all the assets and pipelines for above, including the request POST output for the 2nd pipeline POST against my test server.
There may be some more clever, more elegant plumbing that others can share – please do! I was just focusing on how to get something to work for you (and me, and others).
Do let me know if this helps, or if doesn’t, where you are having problems. Many of our snaps wrap REST API’s so you don’t have to think about these details.
rest-post-1-file-2-form-fields_2021_08_13.slp (3.7 KB)
rest-post-2-files-4-form-fields_2021_08_13.slp (4.6 KB)
rest-post-example-assets.zip (7.7 KB)
08-16-2021 08:01 AM
Hi @mbowen,
Thank you for the response. You have specified the files in rest post snap. It is easy to specify the files in rest post snap, if we know the number of files and their content-types in advance.
But in my case, I don’t have a clue about number of files I receive and their content types. The file number vary from 1 to 10 and their content type also varies. Could you please help me with the request?
08-16-2021 04:44 PM
Tell me a little about the service(s) you’re invoking? Are they custom in-house services that you control, or external? Our REST snap pack is not a premium pack, so while it provides a fairly broad range of functionality it may be a bit rough around the edges for some use cases. What I’m getting at is that it’s not as light weight as curl, where I can see dynamically creating POST requests without too much effort.
I don’t have much experienced with form-related data, so I’m not sure if that could help here.
Representing a form field key/value as a file seems pretty heavy to me, but we’re piggybacking on the multi file upload functionality. Most of the fields in an upload file row are expression enabled, so could be supplied from either input data or pipeline parameters.
If you controlled the backend, I suppose you could pack all non file form field data into one file and then decode that in the backend. Again, not pretty, and I won’t even suggest this. It’s not obvious how to inject an upload file table into this snap.
That said, I’m often amazed at other people’s pipeline solutions. I wonder if any folks are using scripting for this – probably not for file upload, but maybe. Our Script snap supports a variety of languages (Python, JavaScript, …) with limitations though.