Hello, I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below. https://xxxxxx–xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=1000&pageNo=1. I am trying to design a process which loops thru till the REST API call returns null data set. For example, if the above REST URL gives output, I have to call the same URL again with the pageNo input parameter as 2 and if it provides a data set, then 3 and on and on. The loop has to end when the REST API comes out with a null data. Any help to solve this would be highly appreciated and also would like to know if looping is the only option and right approach. Thanks Aravind N

can you share the JSON output of this REST GET without pagination? The best way to design “has next” and “next URL” I found is to use a mapper after REST GET to come up with an expression that will give you true for “has next” and next page URL for “next URL”. Once you figure those out then just copy those expressions back to the REST GET configuration.

You need to follow my suggestion to construct your next URL by using a mapper after your rest get. All those $original.xxxxxx isn’t matching in your provided JSON. E.g. original.url should be original.original.url

Thanks Alan for helping me solve the issue. I really appreciate you taking time to help me solve this.

@sanjaynayak, You can stop the iteration if the response returns which page is the current page. So for example if the response returns something like: { . . currentPage: 9, hasNext: true, . . } Your logic to stop the iteration would be: $entity.response.hasNext == true && $entity.response.currentPage <= 10 Here, the iteration will be from page 1 to page 10 and then the Has next attribute becomes false which means it will stop the iteration.

@tlikarish , yes this is one of the option but server side code is just returning has next=true/false and next token, no page number. But we have resolved it with REST Get itself. has next=true && snap.out.totalCount < 10

How to loop thru a REST snap | SnapLogic - Integration Nation

26 Replies

arvindnsn
Contributor
7 years ago
Thanks Alan for helping me solve the issue. I really appreciate you taking time to help me solve this.
dmiller
Former Employee
7 years ago
Is this is a duplicate of How to Access data from an REST API which has pagination as input parameters?
- arvindnsn
  Contributor
  7 years ago
  It’s in the line of same Topic. But i was trying to see if i can loop thru the REST GET Snap (even using any external code like python).
  - aleung
    Contributor III
    7 years ago
    
    arvindnsn:
    
    to call the same URL again with the pageNo input parameter as 2 and if it provides a data set, then 3 a
    
    The REST GET have a build in pagination feature. Please review Example #4, 5 & 6 in the REST GET documentation.
TimBurns
New Contributor III
6 years ago
This is interesting, but how is it useful if you need to write the results of a REST call out using another Snap?
- tlikarish
  Employee
  6 years ago
  The REST Get snap will output documents, which downstream you could write out to a database, another web service, etc… Could you talk more about what you’re trying to do?
  - sanjaynayak
    New Contributor III
    4 years ago
    Do we have any option to stop iteration after some time to process less volume rather too much.
    
    Example
    Assume Api has 1000 pages and 1000 times has next = true and Next URL is running.
    it is too much collecting 1000 pages document in rest get and processing next snap and we are getting Snap was aborted and java util concurrent issues etc…
    Is there any option to set “has next”= false explicitly, though API is returning true after 10 iteration so that we can process only 10 pages rather than 1000 pages.
TimBurns
New Contributor III
6 years ago
It looks like the Snap just swallows the errors on the callback URL, so it wasn’t creating documents every time it ran. I fixed the URL and now it is.

Forum Discussion

How to loop thru a REST snap

26 Replies

Recent Discussions

Pagination and nextCursor in header

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?