How to loop thru a REST snap

Hello,

I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below.

https://xxxxxx–xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=1000&pageNo=1.

I am trying to design a process which loops thru till the REST API call returns null data set.

For example, if the above REST URL gives output, I have to call the same URL again with the pageNo input parameter as 2 and if it provides a data set, then 3 and on and on. The loop has to end when the REST API comes out with a null data.

Any help to solve this would be highly appreciated and also would like to know if looping is the only option and right approach.

Thanks
Aravind N

Is this is a duplicate of How to Access data from an REST API which has pagination as input parameters?

It’s in the line of same Topic. But i was trying to see if i can loop thru the REST GET Snap (even using any external code like python).

The REST GET have a build in pagination feature. Please review Example #4, 5 & 6 in the REST GET documentation.

Based on the Example 4 of the REST GET documentation, I tried to use the solution but getting errors. Attached is the screenshot of the REST GET snap which is used in my pipeline.

The scenario for my REST GET is mentioned below. The API URL has a header key for authorization as mentioned in the screenshot.

https://xxxxxx–xxx/transactions?datestart=2018-12-16T00:00:00&dateend=2018-12-17T00:00:00&pageSize=1000&pageNo=1

If the above URL has response with data, then the below should be triggered

https://xxxxxx–xxx/transactions?datestart=2018-12-16T00:00:00&dateend=2018-12-17T00:00:00&pageSize=1000&pageNo=2

If the above URL has response with data, then pageNo gets incremented until there is no data returned for the API.

The Has next attribute is set as “$entity.Items.length > 0” as i am checking the condition if the API returns any data. But the expression is failing.

Also the Next URL is set in such a way that it takes the input parameter along with the incremental value for pageno. Does this need to have the Header Authorization Key in the call? Please advise and help.

Thanks
Aravind N

can you share the JSON output of this REST GET without pagination?

The best way to design “has next” and “next URL” I found is to use a mapper after REST GET to come up with an expression that will give you true for “has next” and next page URL for “next URL”. Once you figure those out then just copy those expressions back to the REST GET configuration.

I am able to get the condition which returns true if data exists for “Has next” which is “jsonPath($, “$entity.Items[*]”).length > 0”.

The “Next URL” has the correct URL, but is not triggering the URL as it is loading only the data for the Page = 1. The basic URL has the key at the header level. Does the Next URL attribute need to also have the key mentioned?

Here is the output structure

{
“PageNo”: 5,
“PageSize”: 200,
“Items”: [
{
“TransactionId”: 000000,
“ReceiptNo”: 000,
“TerminalName”: “xxxxx”,
“TerminalNo”: 00000,
“Date”: “2018-12-16T18:07:03”,
“TaxTotal”: 999,
“TransactionItems”: [
{
“ItemID”: 999,
“ProductID”: 999,
“Quantity”: 1,
“Total”: 5,
“TotalGross”: 4.55,
“TotalTax”: 0.45
}
],
“OrderTypeID”: 0
}

Not for HTTP Header. Have you tried to place a static URL for page 2 and see if it works?

I tried the Next URL with static URL to fetch the page 2, but it is not loading after the Page 1 data.

which means your “has Next” is false.

Can you share the entire JSON output from “Download” from the output view?

I have the JSON output attached.

REST_GET For Task Transactions.zip (2.7 KB)

here should be your hasNext expression, assuming each page return maximum of 10 records. There also doesn’t seem to be any other data from your output which indicate if there is a next page. So, all we can do is to check if result has the maximum number of records in return.

$entity.PageSize == 10

As for your next url

the page number should be “$entity.PageNo+1” insterad of “$original.pageno_1”

I tired the above solution in vain. I set hasNext = $entity.PageSize == 10
and set next url as both

$original.uri+"?datemin="+$original.datemin+"&datemax="+$original.datemax+"&pagingArgs.pageSize="+$original.pagesize+"&pagingArgs.pageNo="+($entity.PageNo+1)

and also tried using

$original.uri+"?datemin="+$original.datemin+"&datemax="+$original.datemax+"&pagingArgs.pageSize="+$original.pagesize+"&pagingArgs.pageNo="+($original.pageno+1)

Its only loading the first page and not pulling result from subsequent pages.

You need to follow my suggestion to construct your next URL by using a mapper after your rest get. All those $original.xxxxxx isn’t matching in your provided JSON. E.g. original.url should be original.original.url

1 Like

I tried constructing the URL in the mapper snap next to REST snap and got the Dynamic URL as $original.original.url… I also tried using the actual URL (without using any variables, just plain URL as “https://example-api.com/transactions?datemin=…”

Both failed to fetch data from subsequent pages except the first page. The API for the given input parameters fetches close to 2000 records. i tried setting the page size to 10, 50, 100 and 200. But in all the cases, its fetching the data from the first page for whatever the size i set and comes out and finishes the process.

Thanks Alan for helping me solve the issue. I really appreciate you taking time to help me solve this.