cancel
Showing results for 
Search instead for 
Did you mean: 

HTTP snap pagination Not Working

mohit_jain
New Contributor III

Hi Everyone,

I hope you're doing well. I need some assistance with implementing pagination in SnapLogic for the Google Search Console API. Currently, there's a row limit of 25,000, making it challenging to load the data on a daily basis. If the data exceeds this limit, we risk losing valuable information.

To avoid this, I’m looking for a way to implement a loop within the API calls to handle pagination efficiently. Given the urgency of this task, any guidance or suggestions on how to achieve this would be greatly appreciated.

Here is the snap config

mohit_jain_0-1725451942766.png

now we need to update the startRow to +25000 in every execution and it should be end once it is completed

api result :


mohit_jain_1-1725452176925.png

 

Can anyone help me in this
 

1 ACCEPTED SOLUTION

mohit_jain
New Contributor III

Really Thanks to @koryknick and @Palermo 

We have found one solution and it is working fine 
What we have changed is we have passed all the raw entity in the query parameters 

mohit_jain_0-1725604975156.png

And set the pagination parameters like this and it works 

mohit_jain_1-1725605026364.png

 

But thanks everyone for looking and supporting us on this

View solution in original post

10 REPLIES 10

mohit_jain
New Contributor III

koryknick
Employee
Employee

@mohit_jain - Please refer to the SnapLogic documentation for the HTTP Client under HTTP Client Examples

Examples 1 and 2 should give you an idea how the functionality works to paginate the results.  From the screenshots you provided, I believe you would be able to use the Has Next to tell SnapLogic that there is more to get, which you can probably just check to see if the number of results returned is the full 25000:

$entity.rows.length == 25000

Then enable the "Override parameters" to update the "startRow" value.  This value would probably be something like:

(snap.out.totalCount + 1) * 25000

 Hope this helps!

mohit_jain
New Contributor III

Hi @koryknick 

Really thanks for the reply
But I have tried with the same approach which you have mentioned in the last comment 

mohit_jain_0-1725474150999.png

 

But it is giving me error 

mohit_jain_1-1725474201107.png

I have tried with the second way also 

mohit_jain_2-1725474284108.png

 

But still it is giving me same error 
Can you please help 



koryknick
Employee
Employee

Change Has Next to the following:

$.hasPath('entity.rows') ? $entity.rows.length == 25000 : false

Then you may need to look at the second response and see why $entity.rows doesn't exist - it seems odd that you're getting a different response unless there are exactly 25000 documents available.