How to loop thru a REST snap

[arvindnsn]

Hello,

I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below.

https://xxxxxx–xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=1000&pageNo=1.

I am trying to design a process which loops thru till the REST API call returns null data set.

For example, if the above REST URL gives output, I have to call the same URL again with the pageNo input parameter as 2 and if it provides a data set, then 3 and on and on. The loop has to end when the REST API comes out with a null data.

Any help to solve this would be highly appreciated and also would like to know if looping is the only option and right approach.

Thanks
Aravind N

[sanjaynayak]

Do we have any option to stop iteration after some time to process less volume rather too much.

Example
Assume Api has 1000 pages and 1000 times has next = true and Next URL is running.
it is too much collecting 1000 pages document in rest get and processing next snap and we are getting Snap was aborted and java util concurrent issues etc…
Is there any option to set “has next”= false explicitly, though API is returning true after 10 iteration so that we can process only 10 pages rather than 1000 pages.

[j_angelevski]

@sanjaynayak,

You can stop the iteration if the response returns which page is the current page. So for example if the response returns something like:

{
.
.
currentPage: 9,
hasNext: true,
.
.
}

Your logic to stop the iteration would be:

$entity.response.hasNext == true && $entity.response.currentPage <= 10

Here, the iteration will be from page 1 to page 10 and then the Has next attribute becomes false which means it will stop the iteration.

[sanjaynayak]

Thanks a lot @j.angelevski , this api is just returning has next=true/false along with new page token but it is helpful, will check what other parameter we can use from response to break the loop.

[sanjaynayak]

@j.angelevski , API response side no parameter to terminate after 10 number of iteration.
We should implement another parameter next url call limit along with has next, next url for looping in rest API.

Since server side design various company to company, in this case if it would have to page number concept then it was easy to handle but in this case instead of page number, it has next new token for has next =true.

Alternate way we can use head snap to read first 10 iteration for next processing but rest snap still running until 1000 iteration to be completed.

Is there any other snap we can consider to find some alternate for his.

[tlikarish]

If I understand you correctly, you’re trying to fetch data from an API that has implemented pagination. The API will return a field called hasNext that is either true or false depending on if there is more data. However, you want more control over the pagination as you don’t want to fetch all the data at one time. For example, you may only want to fetch the first 10 pages, instead of the full 1000.

One way to accomplish this is with a Sequence snap, where you can specify you want it to output 10 values, or however many pages you want to fetch. Then downstream use the Rest Get snap.

Here is an example pipeline to try out.

rest-pagination-example_2022_01_04.slp (3.8 KB)