Convert this to a snap? How?

I’m trying to use an API to post data to an application.

There is no account. Instead I have a specific account.

The curl API looks like this:
curl https://communicate.modolabs.net/api/v1/directories/revisions --header “Authorization: Token myToken” --form “name=Modo Test” --form “upload=@directory.xml” where “myToken” would be the token I was given.

How do I convert this to the REST Snap?

Hi Bob:
I will follow-up tomorrow (Wed, 5/19) with an example of how to do this.

Hi Bob:

I’m still working on the example. It’s a little trickier than I thought. I’ve been inspecting and reconciling request payloads. I am closer to matching a curl request against an echo server (shown below). Contents of directory.xml is just arbitrary xml. Hope to have example finished soon. Thanks for your patience.

POST / HTTP/1.1
Authorization: myToken
Accept: */*
User-Agent: curl/7.68.0
Host: 127.0.0.1:3000
Content-Length: 387
Content-Type: multipart/form-data; boundary=------------------------5217129846a80aae

--------------------------5217129846a80aae
Content-Disposition: form-data; name="name"

Modo Test
--------------------------5217129846a80aae
Content-Disposition: form-data; name="upload"; filename="directory.xml"
Content-Type: application/xml

<directory name="foo">
  <file>file1.txt</file>
  <file>file2.json</file>
</directory>

--------------------------5217129846a80aae--

Hi Bob:

Disclaimer: I’m familiar with many of our snaps, but actually haven’t done much with our REST snaps. So, maybe others in the Community will chip in with extra explanation or solutions.

Here is a solution that may work. The pipeline is just a single REST POST snap. You’ll see that the REST snap inserts a few extra attributes that Curl doesn’t which may be ok (ex: Content-Transfer-Encoding: binary).

File: value1.txt

  Mondo Test

File: directory.xml

  <directory name="foo">
    <file>file1.txt</file>
    <file>file2.json</file>
  </directory>

My Service URL points to a local echo server. You’ll see that I encoded the “name” property value into a file – not super pretty, but gets us close to the representation that we want. I uploaded the file (sldb://), but file could also be referenced on file system (file://).

For our “name” property, I put an “x” for the filename to indicate that really don’t care about this. If left blank, it will be substituted with the name of the uploaded file (ie, “value1.txt”).

A couple other observations. Upload Body Type is Multipart FORM-DATA. HTTP entity fields are empty. These fields are ignored for multipart form-data.

Here is the generated request payload.

Accept=[*/*]
Connection=[Keep-Alive]
Host=[127.0.0.1:8001]
Transfer-encoding=[chunked]
Authorization=[Token <my-token>]
Content-type=[multipart/form-data; boundary=Kz659krJ927tbvVX9g5Wwe9siMHfcZOUEqsZvKqr]

--Kz659krJ927tbvVX9g5Wwe9siMHfcZOUEqsZvKqr
Content-Disposition: form-data; name="name"; filename="x"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary

Mondo Test

--Kz659krJ927tbvVX9g5Wwe9siMHfcZOUEqsZvKqr
Content-Disposition: form-data; name="upload"; filename="directory.xml"
Content-Type: application/xml; charset=UTF-8
Content-Transfer-Encoding: binary

<directory name="foo">
  <file>file1.txt</file>
  <file>file2.json</file>
</directory>

--Kz659krJ927tbvVX9g5Wwe9siMHfcZOUEqsZvKqr--

I’m not sure if filename=“x” will cause a problem with submission, but it’s not obvious how to not display this.

Another approach is to submit request as a Multipart RELATED type. This didn’t get me any closer, but thought I would document what I did anyway.

form-related-pipeline

This pipeline includes a Mapper and REST Snap. I defined the form property in json under an “entity” key in a Mapper, and then referenced this variable in the HTTP Entity field of the REST snap.

In the generated request, this part is encoded as json content with a hard-coded “metaData” key. This key name is part of the Multipart RELATED plumbing and is not obvious how to override.

Here is the generated request payload for Multipart RELATED.

Accept=[*/*]
Connection=[Keep-Alive]
Host=[127.0.0.1:8001]
Transfer-encoding=[chunked]
Authorization=[Token <my-token>]
Content-type=[multipart/related; boundary=2_cNQDNeLIqanSgxKib8BNkTrwuamXl]

--2_cNQDNeLIqanSgxKib8BNkTrwuamXl
Content-Disposition: form-data; name="metaData"
Content-Type: application/json; charset=UTF-8
Content-Transfer-Encoding: 8bit

{ "name": "Modo Test" }
--2_cNQDNeLIqanSgxKib8BNkTrwuamXl
Content-Disposition: form-data; name="upload"; filename="directory.xml"
Content-Type: application/xml; charset=UTF-8
Content-Transfer-Encoding: binary

<directory name="foo">
  <file>file1.txt</file>
  <file>file2.json</file>
</directory>

--2_cNQDNeLIqanSgxKib8BNkTrwuamXl--

did you ever solve this?

@mbowen: Did you get a solution? I have a similar requirement where I need to post multiple files to SnapLogic, the number of files and the content type.

I haven’t been able to get this work for my use. Tried different things and nothing worked.

@mike.korcynski @Sowmya_Rayavarapu @robert_parks
That’s horrible news guys. Again, I actually haven’t done a lot of work with our REST snaps and just to post this earlier post involved some learning for me. However, we absolutely should be able to get this to work! Maybe, I posted too many details suggesting I knew exactly how it works. I can roll-up my sleeves and help you guys get this working.

Thank you so much Matthew. Keep us posted.

For me at least, if I understand the problem correctly, the primary issue seems to be when you set a form field to a file. you can’t then continue to set additional form fields.

@mike.korcynski
Not sure on this, but you’re probably right. I will investigate! I likely won’t be able to look into this today, but tomorrow (Thu, 8/12) for sure.

@mike.korcynski @Sowmya_Rayavarapu @robert_parks

Here is a solution to POST-ing a file with multiple form fields.

Our REST POST snap is a swiss-army knife, and I am less familiar with all of its options (multipart related, for example). I need to better understand the “entity” plumbing too. That said, let’s evaluate something that might work – maybe not an awesome solution, but hopefully something that does work.

We want to POST more than one thing, so let’s ignore the “single file upload” fields (4 of them). We’ll leave all the entity fields empty too.

The conceptual model is that we need to create a file for every field we want to submit. If we want to upload a file and also specify two form fields, we’ll create 3 files! Files can be referenced in SLDB, or the filesystem which may be less convenient for you depending on your Snaplex filesystem access.

Here’s the top half of the REST POST snap showing the fields I left empty.

I’ve attached two simple pipelines each with a single REST POST snap.

For the first pipeline we want to upload a file (directory.xml) and also specify two form fields. Here’s the curl version:

curl -X POST http://127.0.0.1:8001/test \
    --form "upload=@directory.xml" \
    --form "field1=field1-value" \
    --form "field2=field2-value"

We can define the form fields as files using the form field name as the “Upload-file key”. We then specify a file which contains the form field’s value. field1.txt stores the string “field1-value”. We also specify the content type as text/plain.

Here’s how the POST is received in my test server.

> POST /test

> REQUEST HEADERS:
Accept=[*/*]
Connection=[Keep-Alive]
Host=[127.0.0.1:8001]
Transfer-encoding=[chunked]
Content-type=[multipart/form-data; boundary=PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2]

> REQUEST BODY:
--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="upload"; filename="directory.xml"
Content-Type: application/xml; charset=UTF-8
Content-Transfer-Encoding: binary

<directory name="foo">
  <file>file1.txt</file>
  <file>file2.json</file>
</directory>

--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="field1"; filename="x"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary

field1-value

--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2
Content-Disposition: form-data; name="field2"; filename="x"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary

field2-value

--PusHivUIyaSW4MgQOpnzZn0wvMu-KHt2--

Ok, for the second pipeline we get a bit more ambitious and double it, that is, upload 2 files, and specify 4 form fields. The second file in this case is an image (a SnapLogic logo, of course!). We also specify some of the form field files on the filesystem (ie, file:///tmp). The 4th form field is defined as json. That’s a nice part of multipart upload, each part defines its content type.

I’ve attached all the assets and pipelines for above, including the request POST output for the 2nd pipeline POST against my test server.

There may be some more clever, more elegant plumbing that others can share – please do! I was just focusing on how to get something to work for you (and me, and others).

Do let me know if this helps, or if doesn’t, where you are having problems. Many of our snaps wrap REST API’s so you don’t have to think about these details.

rest-post-1-file-2-form-fields_2021_08_13.slp (3.7 KB)
rest-post-2-files-4-form-fields_2021_08_13.slp (4.6 KB)
rest-post-example-assets.zip (7.7 KB)

Hi @mbowen,

Thank you for the response. You have specified the files in rest post snap. It is easy to specify the files in rest post snap, if we know the number of files and their content-types in advance.

But in my case, I don’t have a clue about number of files I receive and their content types. The file number vary from 1 to 10 and their content type also varies. Could you please help me with the request?

@Sowmya_Rayavarapu

Tell me a little about the service(s) you’re invoking? Are they custom in-house services that you control, or external? Our REST snap pack is not a premium pack, so while it provides a fairly broad range of functionality it may be a bit rough around the edges for some use cases. What I’m getting at is that it’s not as light weight as curl, where I can see dynamically creating POST requests without too much effort.

I don’t have much experienced with form-related data, so I’m not sure if that could help here.

Representing a form field key/value as a file seems pretty heavy to me, but we’re piggybacking on the multi file upload functionality. Most of the fields in an upload file row are expression enabled, so could be supplied from either input data or pipeline parameters.

If you controlled the backend, I suppose you could pack all non file form field data into one file and then decode that in the backend. Again, not pretty, and I won’t even suggest this. It’s not obvious how to inject an upload file table into this snap.

That said, I’m often amazed at other people’s pipeline solutions. I wonder if any folks are using scripting for this – probably not for file upload, but maybe. Our Script snap supports a variety of languages (Python, JavaScript, …) with limitations though.