Can't paginate using Post HTTP CLIENT Snap

[snapation6713]

I am trying to paginate using a POST HTTP CLIENT Snap.  I don't know how to accomplish this with the information provided in the data returned.  I am extracting data from HubSpot using the 'search' endpoint (https://api.hubapi.com/crm/v3/objects/contact/search).  The data returned looks like this:

"entity":{

"total":

479322

"results":[

{

"id":"123","properties":{...},"createdAt":"2022-06-10T05:49:44.061Z","updatedAt":"2024-05-23T11:58:59.795Z","archived":false

}.....

}

]

"paging":{

"next":{

"after":

"10"

}

How do I use this information to configure the snap to paginate? 

Please Help!

[koryknick]

@snapation6713 - Use the HTTP Client snap in the API Suite snap pack with Request Method = POST.  Then you can configure the Pagination section Has Next property as:

$entity.paging.hasPath('next')

Enable the "Reuse request parameters" and set the key appropriate to the API you're calling for the offset (I assume "after"), and I assume the following expression:

$entity.paging.next.after

This may not all be completely accurate as I can't test it, but hopefully it gets you on your way to a solution.

[snapation6713]

Thank you @koryknick, this is the logical thing to do, which I did try and it returned the first 10 records and an error stating the following:  get the following errors:

---

---

{

"error":

"Failed to execute the request, or the endpoint responded an error"

"stacktrace":

"com.snaplogic.snap.api.SnapDataException: Failed to execute the request, or the endpoint responded an error\n at com.snaplogic.snaps.apisuite.HTTPClient$HttpClientConfig.writeConnectionExceptions(HTTPClient.java:1481)\n at com.snaplogic.snaps.apisuite.HTTPClient.process(HTTPClient.java:375)\n at com.snaplogic.snap.api.ExecutionUtil.process(ExecutionUtil.java:106)\n at com.snaplogic.snap.api.ExecutionUtil.execute(ExecutionUtil.java:70)\n at com.snaplogic.snap.api.SimpleSnap.execute(SimpleSnap.java:70)\n at com.snaplogic.snaps.apisuite.HTTPClient.execute(HTTPClient.java:356)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:801)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeForSuggest(SnapRunnableImpl.java:640)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:857)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:427)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:116)\n at java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n at java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n at java.base/java.lang.Thread.run(Unknown Source)\n"

"reason":

"Got response 400 - Bad Request from the endpoint"

"resolution":

"Refer to the error document in the error view for more information"

"status_code":

"error"

"statusLine":{

"httpVersion":

"HTTP/1.1"

"statusCode":

400

"statusText":

"Bad Request"

}

"headers":{

"Date":"Mon, 10 Jun 2024 15:18:39 GMT","Content-Type":"application/json;charset=utf-8","Content-Length":"179","Connection":"keep-alive","CF-Ray":"891a55cd2967b9d3-SEA","CF-Cache-Status":"DYNAMIC","Strict-Transport-Security":"max-age=31536000; includeSubDomains; preload","Vary":"origin, Accept-Encoding","Access-Control-Allow-Credentials":"false","X-Content-Type-Options":"nosniff"

}

"entity":{

"status":

"error"

"message":

"Invalid JSON input: a request body was expected, but none found"

"correlationId":

"755d6bba-058b-4a4c-a31f-d9694a17cb5b"

"category":

"VALIDATION_ERROR"

}

It seems its can't do the subsequent call for failure to read the body.  

My snap is configured as follows:

[snapation6713]

In addition when I look at the results the debug section contains the following: 

"entity":

"[UnrepeatableEntity] {Content-Type: Content-Type: application/json;charset=utf-8,Content-Length: 359,Chunked: false}"

Not sure what makes the entity 'unrepeatable'

[koryknick]

@snapation6713 - my apologies - I didn't see that you had referenced the API you're trying to call.  Fortunately, I recently worked with the Hubspot API and that one is a bit tricky as it uses a non-standard method to paginate their data.  Please refer to our HTTP Client Pagination documentation, which will give clear instructions that should work for this particular API framework.

[snapation6713]

@koryknick I did see this solution before and I tried it.  I downloaded the sample pipeline and had to make some modifications to the json body by removing the json.stringify and unselecting the '=' and add a header of content-type: application/json and I did get many results and no errors, however the issue here is that it returns a limit of 10 (even though I have the limit set to 100)   and it returns the same number of total records and the same  first 10 records for all the pages.

This is the debug:

debug":{

"requestString":

"POST /crm/v3/objects/contact/search?page=3&per_page=100 HTTP/1.1 [Accept: */*, Content-Type: application/json, Authorization: <--Hidden-->, Content-Length: 224, Host: api.hubapi.com, Connection: Keep-Alive, User-Agent: snaplogic/main26341 httpcore/37.0 (11.0.17), Accept-Encoding: gzip,deflate] [{\n "filters": [\n {\n "propertyName": "lastmodifieddate",\n "operator": "GT",\n "value": "1900-01-01T00:00:00.000Z"\n }\n ],\n"sort":[{"field":"lastmodifieddate",\n"order":"asc"}]\n}]"

"request":{

"requestLine":

"POST /crm/v3/objects/contact/search?page=3&per_page=100 HTTP/1.1"

"headers":[

{

"Accept":"*/*"

}

{

"Content-Type":"application/json"

}

{

"Authorization":"<--Hidden-->"

}

{

"Content-Length":"224"

}

{

"Host":"api.hubapi.com"

}

{

"Connection":"Keep-Alive"

}

{

"User-Agent":"snaplogic/main26341 httpcore/37.0 (11.0.17)"

}

{

"Accept-Encoding":"gzip,deflate"

}

]

"timeStamp":{

"Date":"2024-06-10T19:27:25.960+0000"

}

"entity":

"{\n "filters": [\n {\n "propertyName": "lastmodifieddate",\n "operator": "GT",\n "value": "1900-01-01T00:00:00.000Z"\n }\n ],\n"sort":[{"field":"lastmodifieddate",\n"order":"asc"}]\n}"

}