Adding increments to a REST Get paging request

[DMG]

We have to send GET requests to a REST API which unfortunately doesn't support standard paging. The endpoint returns only 50 records at a time, and does not include in the response any information on whether there are more records (such as a 'hasMore' field) or what is their current offset. All that is included in the response is an "entity" array with maximum 50 records.

However, the REST API does support adding an "offset" value to the GET request, but the problem is that the response won't include the information of the offset it was requested in the first place.

In the "Has next" condition field of my REST snap, I can check the size of the entity array and set the condition to  trigger the next request when $entity.length == 50. 

If I set the "Next URL" as "https://<base_url>/api/records?offset=50" the second GET request will retrieve records from 51 to 100, but so will the third GET request and so on, and it will basically loop forever on the second page.

I need a way to add increments of 50 to the offset. If I knew the offset which was requested the previous time, or the number of the current iteration, then I could use either of them in the "Next URL" formula, but is this possible at all? Is there a way from within the "Next URL" formula to access the previous value of "Next URL" itself?

[fabriciocarboni]

Hi!

Coincidence or not I have just answered this question here

[koryknick]

@DMG @fabriciocarboni - I recommend using the HTTP Client snap in the API Client snap pack.  You can use the Pagination section to test if there is more to do based on the API results (i.e. number of returned records is less than the expected page size).  Then use the "Reuse request parameters" to provide the next offset.  You can use the built-in snap.out.totalCount to assist with this expression, which would probably be something like:

Hope this helps!