SnapLogic - Integration Nation

snapation6713 · 3 weeks ago

I am trying to paginate using a POST HTTP CLIENT Snap. I don't know how to accomplish this with the information provided in the data returned. I am extracting data from HubSpot using the 'search' endpoint (https://api.hubapi.com/crm/v3/objects/contact/search). The data returned looks like this:

"entity":{

"total":

479322

"results":[

{

"id":"123","properties":{...},"createdAt":"2022-06-10T05:49:44.061Z","updatedAt":"2024-05-23T11:58:59.795Z","archived":false

}.....

}

]

"paging":{

"next":{

"after":

"10"

}

How do I use this information to configure the snap to paginate?

Please Help!

snapation6713 · a week ago

@koryknick, Thank you for your help. I was provided a solution by a coworker.

It's essentially that pipeline solution you provided HTTP Client Pagination with a few tweaks. I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap "after" : $splitValue, "limit" : 100 Then I was getting subsequent data. But the data was duplicating. So I modified the map for pagination as follows:

sl.range(1, parseInt($entity.total), 100) -I also changed the 1st http snap to just give me 1 record just so that I get the total and I process from page 1 to the end in the rest of the pipeline so I don't do a union.

And it worked! I am now getting subsequent data and it the records are not duplicating.

View solution in original post

fabriciocarboni · 2 weeks ago

Hi,

I had the exactly same problem and could not find a way to do that. The documentation is poor. What I had to do was develop a python script to achive this to load lots of historical data with pagination because the api allowed me only to retrieve 1000 records at a time.

snapation6713 · 2 weeks ago

Thank you for taking the time to reply @fabriciocarboni. Did you create the python script in snaplogic? If so did you have to install any python packages to get it to execute?

fabriciocarboni · 2 weeks ago

Hi @snapation6713,

No, our Snaplogic subscription does not have RPE. So I had to develop the script and run it from my local machine.

snapation6713 · 2 weeks ago

Ah! Ok thank you so much for your time!

Enjoy your day.

SnapLogic - Integration Nation

Can't paginate using Post HTTP CLIENT Snap