This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
FAQ
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

REST Get Pagination in various scenarios

Vishaal_Arun
New Contributor II

2 weeks ago - last edited 2 weeks ago

Hi all,

There are various challenges witnessed while using REST GET pagination. In this article, we can discuss about these challenges and how to overcome these challenges with the help of some in-build expressions in snaplogic. Let's see the various scenarios and their solution.

Scenario 1: API URL response has no total records indicator, but works with limit and offset:

In this case, as there are no total records that the API is going to provide in advance the only way is to navigate each of the page until the last page of the API response. The last page of the API is the page where there are no records in the response output.

Explanation of how it works and Sample data:
has_next condition:
      $entity.length > 0

has_next explanation:
             If the URL has n documents and it is not sure if the next page iteration is valid, the function $entity.length will check the response array length from the URL output and proceeds with the next page iteration only when the $entity.length is greater than zero. If the response array length is equal to zero, it’s evident that there are no more records to be fetched and hence the condition on has_next “$entity.length > 0” will fail and stops the next iteration loop.

next_url condition:
            $original.URL+"?limit=" + $original.limit + "&offset=" + ( parseInt($original.limit) * snap.out.totalCount )

next_url explanation:
             Limit (limit parameter) and API URL values are static, but the offset value will need to change for each iteration. Hence the approach is to multiply the default limit parameter (limit) with the snap.out.totalCount function to shift the offset per API page iteration. snap.out.totalCount is the snap system variable which used to hold the total number of documents that have passed through output views of the snap. In this “REST Get”, each API page iteration response output is one json array and hence the snap.out.totalCount will be equal to the number of API page iteration completed

Sample response:

 

 

For First API call: 
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Mark",
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "John",
        },………………. 1000 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
        "limit": "1000",
        "offset": "0",
           "URL": "https://Url.XYZ.com"
    }
}

For Second API Call:
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2024",
            "month": "08",
            "Name": "Ram",
        },
        {
            "year": "2021",
            "month": "03",
            "Name": "Joe",
        },………………. 1000 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
        "limit": "1000",
        "offset": "1000" 
          "URL": "https://Url.XYZ.com"
    }
}

 

 

Scenario 2: API URL response has total records in the response header and pagination is using limit & offset:

As there are total records, the total records column in the API response can be used to traverse through the API response pages.

Explanation on how it works and Sample data:
has_next condition:
         parseInt($original.limit) * snap.out.totalCount < $headers[total-records]

has_next condition explanation:
        If the URL Response has n documents where n is equal to total, there needs a check whether the limit is less than total records, for example: if there were 120 total records and 100 as a limit, it loops through only 2 times.

It loops through as below,
limit = 100, snap.out.totalCount =0:
has_next condition will evaluate 0 < 120
limit = 100, snap.out.totalCount =1
has_next condition will evaluate 100 < 120
limit = 100, snap.out.totalCount =2
has_next condition will evaluate 200 < 120 pagination breaks and next page is not processed

next_url condition:
       $original.URL+"?limit=" + $original.limit + "&offset=" + (parseInt($original.limit)* snap.out.totalCount)

next_url Explanation:
         Limit and url values are static, but the offset value need to be derived as limit multiplied with snap.out.totalCount function. snap.out.totalCount indicates the total number of documents that have passed through all of the Snap's output views. So it will traverse next API page until the has_next condition is satisfied

Sample Response:

 

 

 For First API call: 
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Mark"
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "John"
        },….....100 records
    ],
    "original": {
        "effective_date": "2023-08-31",
        "limit": "100",
        "offset": "0",
          "URL": "https://Url.XYZ.com"
    } 
    "headers": {
        "total-records": [
            "120"
        ]
    }
}

For Second API Call:
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Ram"
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "Raj"
        },….....20 records
    ],
    "original": {
        "effective_date": "2023-08-31",
        "limit": "100",
        "offset": "100",
           "URL": "https://Url.XYZ.com"
    } 
    "headers": {
        "total-records": [
            "120"
        ]
    }
}

 

 

 

Scenario 3: API has no total records indicator and pagination is using page_no:

The scenario here is that, there is no total records indication in the API output but API has page number as parameter. So the API pagination is possible by incrementing the page number parameter by 1 until the length of the API output array length is greater than 0, else the pagination loop need to break.

Explanation on how it works and Sample data:

Has-next condition:
        $entity.length > 0

Has-next Condition Explanation:
       As there is no total record count known from API output, next page of the API need to be fetched if the current page has any output elements in the output array.

next-url condition:
    $original.URL+"&page_no= " + $headers.page_no+1

Next-Url Condition Explanation:
          As every document has page number in it, same can be used in the has-next condition.

Sample Response:

 

 

 For First API call: 
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Mark",
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "John",
        },………………. 1000 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
           "URL": "https://Url.XYZ.com"
    },
        "headers": {
        "page_no": 1
    }
}


 For Second API call: 
{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Ram",
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "Raj",
        },………………. 1000 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
           "URL": "https://Url.XYZ.com"
    },
        "headers": {
        "page_no": 2
    }
}

 

 

Scenario 4: has total records in the response header and pagination is using page_no

The scenario is there is a total records count indicator and page number in the API Url response. API next page traverse can be through incrementing page number by 1 and validate if the total records count is less than the total rows fetched so far (multiplication of snap.out.totalCount and page limit).

Explanation on how it works and Sample data:

Has_next condition:
      parseInt($original.limit) * snap.out.totalCount < $headers[total-records]

Has-next Explanation:
           If the URL Response has n documents where n is equal to total, has_next condition is to check whether the rows fetched is less than total records, For example: if there were have 120 total records and 100 as the limit factor for the API (predefined as part of design/implementation), it loops through exactly 2 times (first and second page only).

it loops through as below,
limit = 100, snap.out.totalCount =0:
has_next condition will evaluate 0 < 120
limit = 100, snap.out.totalCount =1
has_next condition will evaluate 100 < 120
limit = 100, snap.out.totalCount =2
has_next condition will evaluate 200 < 120 pagination breaks and next page is not processed

Next-url condition:
       $original.URL+"&page_no= " + $headers.page_no+1

next-url explanation:
              As every API URL output has page number in it, same can be used in the has-next condition and also in incrementing page number to get to the next document.

Sample Response:

 

 

{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Mark",
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "John",
        },………………. 100 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
        "in_limit": "100",
           "URL": "https://Url.XYZ.com"
    },
        "headers": {
        "page_no": 1
    }
}

 For Second API call: 

{
    "statusLine": {
        "protoVersion": "HTTP/1.1",
        "statusCode": 200,
        "reasonPhrase": "OK"
    },
    "entity": [
        {
            "year": "2022",
            "month": "08",
            "Name": "Ram",
        },
        {
            "year": "2022",
            "month": "08",
            "Name": "Raja",
        },………………. 20 records in this array
    ],
    "original": {
        "effective_date": "2023-08-31",
        "limit": "100",
           "URL": "https://Url.XYZ.com"
    },
        "headers": {
        "page_no": 2
    }
}

 

 

Please give us Kudos if the article helps you😍

1 REPLY 1

gkumarSnapCommu
Employee
Employee

2 weeks ago

@Vishaal_Arun this is great.Thanks for sharing. Have you also tried out HTTP Client Snap as we have added enhanced features for supporting pagination

0 Kudos