Forum Discussion

snapation6713's avatar
snapation6713
New Contributor III
2 years ago
Solved

Can't paginate using Post HTTP CLIENT Snap

I am trying to paginate using a POST HTTP CLIENT Snap.  I don't know how to accomplish this with the information provided in the data returned.  I am extracting data from HubSpot using the 'search' endpoint (https://api.hubapi.com/crm/v3/objects/contact/search).  The data returned looks like this:

"entity":{
"total":
479322
"results":[
{
"id":"123","properties":{...},"createdAt":"2022-06-10T05:49:44.061Z","updatedAt":"2024-05-23T11:58:59.795Z","archived":false
}.....
 
}
]
"paging":{
"next":{
"after":
"10"
}

How do I use this information to configure the snap to paginate? 

Please Help!

Testing-Deployment-Automation
  • snapation6713's avatar
    snapation6713
    2 years ago

    koryknick,   Thank you for your help.  I was provided a solution by a coworker.

    It's essentially that pipeline solution you provided HTTP Client Pagination  with a few tweaks.  I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap   "after" : $splitValue, "limit" : 100    Then I was getting subsequent data.   But the data was duplicating.  So I modified the map for pagination as follows:  

    sl.range(1, parseInt($entity.total), 100)  -I also changed the 1st http snap to just give me 1 record just so that I get the total  and I process from page 1 to the end in the rest of the pipeline so I don't do a union.  
     
    And it worked!   I am now getting subsequent data and it the records are not duplicating.  

     

     

12 Replies

  • fabriciocarboni's avatar
    fabriciocarboni
    New Contributor II
    2 years ago

    Hi,

    I had the exactly same problem and could not find a way to do that. The documentation is poor. What I had to do was develop a python script to achive this to load lots of historical data with pagination because the api allowed me only to retrieve 1000 records at a time.

    • snapation6713's avatar
      snapation6713
      New Contributor III
      2 years ago

      Thank you for taking the time to reply fabriciocarboni.   Did you create the python script in snaplogic?   If so did you have to install any python packages to get it to execute?

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    snapation6713 - my apologies - I didn't see that you had referenced the API you're trying to call.  Fortunately, I recently worked with the Hubspot API and that one is a bit tricky as it uses a non-standard method to paginate their data.  Please refer to our HTTP Client Pagination documentation, which will give clear instructions that should work for this particular API framework.

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    snapation6713 - Use the HTTP Client snap in the API Suite snap pack with Request Method = POST.  Then you can configure the Pagination section Has Next property as:

     

    $entity.paging.hasPath('next')

     

    Enable the "Reuse request parameters" and set the key appropriate to the API you're calling for the offset (I assume "after"), and I assume the following expression:

     

    $entity.paging.next.after

     

    This may not all be completely accurate as I can't test it, but hopefully it gets you on your way to a solution.

    • snapation6713's avatar
      snapation6713
      New Contributor III
      2 years ago

      Thank you koryknick, this is the logical thing to do, which I did try and it returned the first 10 records and an error stating the following:  get the following errors:

       

      {
      "error":
      "Failed to execute the request, or the endpoint responded an error"
       
      "stacktrace":
      "com.snaplogic.snap.api.SnapDataException: Failed to execute the request, or the endpoint responded an error\n at com.snaplogic.snaps.apisuite.HTTPClient$HttpClientConfig.writeConnectionExceptions(HTTPClient.java:1481)\n at com.snaplogic.snaps.apisuite.HTTPClient.process(HTTPClient.java:375)\n at com.snaplogic.snap.api.ExecutionUtil.process(ExecutionUtil.java:106)\n at com.snaplogic.snap.api.ExecutionUtil.execute(ExecutionUtil.java:70)\n at com.snaplogic.snap.api.SimpleSnap.execute(SimpleSnap.java:70)\n at com.snaplogic.snaps.apisuite.HTTPClient.execute(HTTPClient.java:356)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:801)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeForSuggest(SnapRunnableImpl.java:640)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:857)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:427)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:116)\n at java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n at java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n at java.base/java.lang.Thread.run(Unknown Source)\n"
       
      "reason":
      "Got response 400 - Bad Request from the endpoint"
       
      "resolution":
      "Refer to the error document in the error view for more information"
       
      "status_code":
      "error"
       
       
      "statusLine":{
      "httpVersion":
      "HTTP/1.1"
       
      "statusCode":
      400
       
      "statusText":
      "Bad Request"
       
      }
       
      "headers":{
      "Date":"Mon, 10 Jun 2024 15:18:39 GMT","Content-Type":"application/json;charset=utf-8","Content-Length":"179","Connection":"keep-alive","CF-Ray":"891a55cd2967b9d3-SEA","CF-Cache-Status":"DYNAMIC","Strict-Transport-Security":"max-age=31536000; includeSubDomains; preload","Vary":"origin, Accept-Encoding","Access-Control-Allow-Credentials":"false","X-Content-Type-Options":"nosniff"
      }
       
      "entity":{
      "status":
      "error"
       
      "message":
      "Invalid JSON input: a request body was expected, but none found"
       
      "correlationId":
      "755d6bba-058b-4a4c-a31f-d9694a17cb5b"
       
      "category":
      "VALIDATION_ERROR"
       
      }
      It seems its can't do the subsequent call for failure to read the body.  
       
      My snap is configured as follows:

       

      • snapation6713's avatar
        snapation6713
        New Contributor III
        2 years ago
        In addition when I look at the results the debug section contains the following: 
         
        "entity":
        "[UnrepeatableEntity] {Content-Type: Content-Type: application/json;charset=utf-8,Content-Length: 359,Chunked: false}"
         
        Not sure what makes the entity 'unrepeatable'
  • snapation6713's avatar
    snapation6713
    New Contributor III
    2 years ago

    koryknick I did see this solution before and I tried it.  I downloaded the sample pipeline and had to make some modifications to the json body by removing the json.stringify and unselecting the '=' and add a header of content-type: application/json and I did get many results and no errors, however the issue here is that it returns a limit of 10 (even though I have the limit set to 100)   and it returns the same number of total records and the same  first 10 records for all the pages.

     

    This is the debug:

    debug":{
    "requestString":
    "POST /crm/v3/objects/contact/search?page=3&per_page=100 HTTP/1.1 [Accept: */*, Content-Type: application/json, Authorization: <--Hidden-->, Content-Length: 224, Host: api.hubapi.com, Connection: Keep-Alive, User-Agent: snaplogic/main26341 httpcore/37.0 (11.0.17), Accept-Encoding: gzip,deflate] [{\n "filters": [\n {\n "propertyName": "lastmodifieddate",\n "operator": "GT",\n "value": "1900-01-01T00:00:00.000Z"\n }\n ],\n"sort":[{"field":"lastmodifieddate",\n"order":"asc"}]\n}]"
     
     
    "request":{
    "requestLine":
    "POST /crm/v3/objects/contact/search?page=3&per_page=100 HTTP/1.1"
    "headers":[
    {
    "Accept":"*/*"
    }
    {
    "Content-Type":"application/json"
    }
    {
    "Authorization":"<--Hidden-->"
    }
     
    {
    "Content-Length":"224"
    }
     
    {
    "Host":"api.hubapi.com"
    }
     
    {
    "Connection":"Keep-Alive"
    }
     
    {
    "User-Agent":"snaplogic/main26341 httpcore/37.0 (11.0.17)"
    }
     
    {
    "Accept-Encoding":"gzip,deflate"
    }
    ]
     
    "timeStamp":{
    "Date":"2024-06-10T19:27:25.960+0000"
    }
    "entity":
    "{\n "filters": [\n {\n "propertyName": "lastmodifieddate",\n "operator": "GT",\n "value": "1900-01-01T00:00:00.000Z"\n }\n ],\n"sort":[{"field":"lastmodifieddate",\n"order":"asc"}]\n}"
     
    }

     

  • snapation6713's avatar
    snapation6713
    New Contributor III
    2 years ago

    koryknick,   Thank you for your help.  I was provided a solution by a coworker.

    It's essentially that pipeline solution you provided HTTP Client Pagination  with a few tweaks.  I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap   "after" : $splitValue, "limit" : 100    Then I was getting subsequent data.   But the data was duplicating.  So I modified the map for pagination as follows:  

    sl.range(1, parseInt($entity.total), 100)  -I also changed the 1st http snap to just give me 1 record just so that I get the total  and I process from page 1 to the end in the rest of the pipeline so I don't do a union.  
     
    And it worked!   I am now getting subsequent data and it the records are not duplicating.