Pagination in HTTP Client snap

[Pakhi]

Hi All,
I am working with HTTP client snap and this is my first snap in the pipeline and i am using GET method to read data from API.
I have to make sure data from all pages gets captured from source and for that i have to use pagination.

Can anyone help me in this as i have never used this function.

[Aleksandar_A]

Hello @Pakhi,

It would be helpful if you can provide us with response sample, so we can determine what are going to be the properties on which we will set up the pagination.

In the meantime you can check the offical HTTP Client Snap documentation.
https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2614591489/HTTP+Client#Pagination

BR,
Aleksandar.

[jhipp]

I’m in a similar position and can’t seem to figure out how to iterate through all of the pages with the information I get from the API. The API I’m calling is very barebones, not including the total number from the call or page number. Here’s a example response:

[
    {
        "enrollment_id": 1223524,
        "content_type": "Store Purchase",
        "module_name": "Module 1",
        "user": {
            "id": 111111,
            "first_name": "TEST",
            "last_name": "PERSON 1",
            "email": "example@test.com"
        },
        "campaign_name": "Campaign 1",
        "enrollment_date": "2023-02-15T15:10:57.000Z",
        "start_date": null,
        "completion_date": null,
        "status": "Not Started",
        "time_spent": 0,
        "policy_acknowledged": false
    },
    {
        "enrollment_id": 1223525,
        "content_type": "Store Purchase",
        "module_name": "Module 1",
        "user": {
            "id": 111112,
            "first_name": "TEST",
            "last_name": "PERSON 2",
            "email": "example@test.com"
        },
        "campaign_name": "Campaign 1",
        "enrollment_date": "2023-02-15T15:10:57.000Z",
        "start_date": "2023-02-18T19:43:31.000Z",
        "completion_date": "2023-02-18T20:08:04.000Z",
        "status": "Passed",
        "time_spent": 1574,
        "policy_acknowledged": false
    }
]

Here are my original Query Parameters:

But the problem is I can’t figure out how to increment the page parameter in the pagination section. I’ve figured out how to trigger pagination via the $entity.length comparison, but not how to grab the next page:

Any help is greatly appreciated.

[ptaylor]

How do you know there are more pages to request? I’m only seeing two results in the first page, but you specified 500 results per page. Doesn’t that mean there are no more results?

[jhipp]

I truncated the results for simplicity to show there’s no total_results or pages in my response. I know there are 3 pages of data (~1300 results) because I’ve run API request through postman.

To answer your first question, the only way I know there’s more to request is if there are 500 results returned on a page (max defined by the API), then I’d try to pull the next page. Rinse and repeat until the results returned is not 500. I realize there’s a potential for a campaign to have a number of results divisible by 500 and will make a call that will result in 0 responses.