REST GET: Has Next And Next URL

[jhagist]

I can’t find any good documentation on how to set this up. I have reviewed this article and I am still stuck

https://www.snaplogic.com/blog/rest-get-and-the-snaplogic-public-apis

The json returns the values below. I need to increment the page by one until I get all the values.

This is the pipeline I have set up.

In the mapper I do this: (Based on article link above)

Here is the REST GET Snap

I just don’t know how to get these variable back to the Rest GET

[bhupender_singh]

@jhagist
The logic for has next is to evaluate this expression and if it’s true then make a new get request using value in next url field. Assuming your every request gets you 1000 records in one page. So if you have more than 1000 records then you have to make another request by incrementing the page parameter by 1. According to the given example, 369 requests will bring 369000 records. To get the remaining number of records, 397th get request will be required and $entity.get(‘pageSize’) will return 822 records and this condition will fail. I hope this helps.

1. has next = $entity.get(‘pageSize’) == 1000

2. next url = “https//secure.p03.eloqua.com/API/rest/2.0/data/accounts?count=1000&page=” + str(snap.out.totalCount+2)

[robert_parks]

@jhagist & @bhupender.singh

Hi James and Bhupender,

James – were you able to get this to work?
We are running into the same situation.
Can you confirm that the “has next” can take values that were returned by the initial get?

Our api returns entity.pageNumber & entity.totalPages. Our thought was to have this for the “has next”:
$entity.pagerNumber < $entity.totalPages.

This should be valid, right?

Also, for the next url we were going to use “$entity.pageNumber + 1” to set the page requirement. Is this valid?

Any direction you can provide would be great. I’ll provide some screen shots if needed.
Thanks!

Bob

[jhagist]

Sorry for not finishing this one out.

You are correct. The “has next” must pull it’s values from the API response.

I believe that $entity.pagerNumber < $entity.totalPages. should work for you. To test it put a mapper after the Rest Snap to see what results you are getting.

This should work too. “Also, for the next url we were going to use “$entity.pageNumber + 1” to set the page requirement. Is this valid?” once again add this to the mapper to see if your logic is correct.

This is a screen shot of my working snap. Some of it is parameters or variables captured earlier in the pipeline but you should get the idea.

One note. “Next URL”. You will notice that I have added variables to it. I mistakenly thought the ones set above would apply but that is not the case. You need to rebuild it manually .

FYI: This is an example from Eloqua. The out of the box does not work for the bulk API.

[zotydelamota]

The idea of using a mapper in front of the REST GET Snap is to be able to see the values the entity returns and which you shal use back in the REST GET snap for the fields of:

1. Has next: This should be a boolean expression that will allow for the REST GET Snap to continue looping until expression is false

2. Next url: This should also be an expression that evaluates into the next URL to use with the REST GET call.

Now, one must know that any API REST GET call one makes to a server for data with pagination, should return some type of metadata that will allow you to use in the REST GET Snap to use in your Has next and Next url. In some cases the Next url is also returned and so all one will need to use is the parameter name.

If you would have any questions, please feel free to reach me at crovoskososky.de.la.mota@pwc.com