09-04-2024 05:17 AM - edited 09-04-2024 05:18 AM
Hi Everyone,
I hope you're doing well. I need some assistance with implementing pagination in SnapLogic for the Google Search Console API. Currently, there's a row limit of 25,000, making it challenging to load the data on a daily basis. If the data exceeds this limit, we risk losing valuable information.
To avoid this, I’m looking for a way to implement a loop within the API calls to handle pagination efficiently. Given the urgency of this task, any guidance or suggestions on how to achieve this would be greatly appreciated.
Here is the snap config
now we need to update the startRow to +25000 in every execution and it should be end once it is completed
api result :
Can anyone help me in this
Solved! Go to Solution.
09-05-2024 11:44 PM
Really Thanks to @koryknick and @Palermo
We have found one solution and it is working fine
What we have changed is we have passed all the raw entity in the query parameters
And set the pagination parameters like this and it works
But thanks everyone for looking and supporting us on this
09-05-2024 09:42 AM
or
$entity.get("rows",[]).length == 25000
But the pagination will not work because this is a POST request with data:
According to the documentation, the HTTP Client Snap does not automatically resend data in subsequent pagination requests. But the Google Search Console API requires that for each page. So, the second request returns nothing.
The first request:
The second request:
@koryknick Is there a way to pass HTTP entity(data) for subsequent pagination requests?
09-05-2024 11:44 PM
Really Thanks to @koryknick and @Palermo
We have found one solution and it is working fine
What we have changed is we have passed all the raw entity in the query parameters
And set the pagination parameters like this and it works
But thanks everyone for looking and supporting us on this
09-06-2024 04:34 AM
@mohit_jain - I missed the request body / POST action. Typically pagination is performed using GET so that is the methodology I provided. In case you encounter this again and cannot provide the body in the query parameters, we do have a recommended pattern for paginating POST requests with body content.
I'm happy you found a solution and thank you for posting it back to the page. This is helpful for others with similar questions!
09-06-2024 05:43 AM
@koryknick the pattern doesn't work for the Google Search Console API because it doesn't return 'total' or similar values to determine how many pages are available. Is it possible to request adding an option(checkbox) to HTTP client to allow resending a request body using pagination for POST?
09-17-2024 05:31 AM
@Palermo - sorry for the slow response on this question - I have been out of office. There is a request already created to pass the POST body content to subsequent calls, which is currently under review to be added to the roadmap.