cancel
Showing results for 
Search instead for 
Did you mean: 

Can't paginate using Post HTTP CLIENT Snap

snapation6713
New Contributor III

I am trying to paginate using a POST HTTP CLIENT Snap.  I don't know how to accomplish this with the information provided in the data returned.  I am extracting data from HubSpot using the 'search' endpoint (https://api.hubapi.com/crm/v3/objects/contact/search).  The data returned looks like this:

"entity":{
"total":
479322
"results":[
{
"id":"123","properties":{...},"createdAt":"2022-06-10T05:49:44.061Z","updatedAt":"2024-05-23T11:58:59.795Z","archived":false
}.....
 
}
]
"paging":{
"next":{
"after":
"10"
}

How do I use this information to configure the snap to paginate? 

Please Help!

1 ACCEPTED SOLUTION

snapation6713
New Contributor III

@koryknick,   Thank you for your help.  I was provided a solution by a coworker.

It's essentially that pipeline solution you provided HTTP Client Pagination  with a few tweaks.  I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap   "after" : $splitValue, "limit" : 100    Then I was getting subsequent data.   But the data was duplicating.  So I modified the map for pagination as follows:  

sl.range(1, parseInt($entity.total), 100)  -I also changed the 1st http snap to just give me 1 record just so that I get the total  and I process from page 1 to the end in the rest of the pipeline so I don't do a union.  
 
And it worked!   I am now getting subsequent data and it the records are not duplicating.  
snapation6713_1-1718120216662.png

 

 

View solution in original post

12 REPLIES 12

fabriciocarboni
New Contributor II

Hi,

I had the exactly same problem and could not find a way to do that. The documentation is poor. What I had to do was develop a python script to achive this to load lots of historical data with pagination because the api allowed me only to retrieve 1000 records at a time.

Thank you for taking the time to reply @fabriciocarboni.   Did you create the python script in snaplogic?   If so did you have to install any python packages to get it to execute?

Hi @snapation6713,

No, our Snaplogic subscription does not have RPE. So I had to develop the script and run it from my local machine.

 

 

Ah!  Ok thank you so much for your time!

Enjoy your day.