How to loop thru a REST snap

It looks like the Snap just swallows the errors on the callback URL, so it wasn’t creating documents every time it ran. I fixed the URL and now it is.

Do we have any option to stop iteration after some time to process less volume rather too much.

Assume Api has 1000 pages and 1000 times has next = true and Next URL is running.
it is too much collecting 1000 pages document in rest get and processing next snap and we are getting Snap was aborted and java util concurrent issues etc…
Is there any option to set “has next”= false explicitly, though API is returning true after 10 iteration so that we can process only 10 pages rather than 1000 pages.


You can stop the iteration if the response returns which page is the current page. So for example if the response returns something like:

currentPage: 9,
hasNext: true,

Your logic to stop the iteration would be:

$entity.response.hasNext == true && $entity.response.currentPage <= 10

Here, the iteration will be from page 1 to page 10 and then the Has next attribute becomes false which means it will stop the iteration.

Thanks a lot @j.angelevski , this api is just returning has next=true/false along with new page token but it is helpful, will check what other parameter we can use from response to break the loop.

@j.angelevski , API response side no parameter to terminate after 10 number of iteration.
We should implement another parameter next url call limit along with has next, next url for looping in rest API.

Since server side design various company to company, in this case if it would have to page number concept then it was easy to handle but in this case instead of page number, it has next new token for has next =true.

Alternate way we can use head snap to read first 10 iteration for next processing but rest snap still running until 1000 iteration to be completed.

Is there any other snap we can consider to find some alternate for his.

If I understand you correctly, you’re trying to fetch data from an API that has implemented pagination. The API will return a field called hasNext that is either true or false depending on if there is more data. However, you want more control over the pagination as you don’t want to fetch all the data at one time. For example, you may only want to fetch the first 10 pages, instead of the full 1000.

One way to accomplish this is with a Sequence snap, where you can specify you want it to output 10 values, or however many pages you want to fetch. Then downstream use the Rest Get snap.

Here is an example pipeline to try out.

rest-pagination-example_2022_01_04.slp (3.8 KB)

@tlikarish , yes this is one of the option but server side code is just returning has next=true/false and next token, no page number. But we have resolved it with REST Get itself.

has next=true && snap.out.totalCount < 10

1 Like