โ02-16-2023 09:52 AM
Hi All,
I am working with HTTP client snap and this is my first snap in the pipeline and i am using GET method to read data from API.
I have to make sure data from all pages gets captured from source and for that i have to use pagination.
Can anyone help me in this as i have never used this function.
โ02-17-2023 12:13 AM
Hello @Pakhi,
It would be helpful if you can provide us with response sample, so we can determine what are going to be the properties on which we will set up the pagination.
In the meantime you can check the offical HTTP Client Snap documentation.
https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2614591489/HTTP+Client#Pagination
BR,
Aleksandar.
โ02-23-2023 09:52 AM
Iโm in a similar position and canโt seem to figure out how to iterate through all of the pages with the information I get from the API. The API Iโm calling is very barebones, not including the total number from the call or page number. Hereโs a example response:
[
{
"enrollment_id": 1223524,
"content_type": "Store Purchase",
"module_name": "Module 1",
"user": {
"id": 111111,
"first_name": "TEST",
"last_name": "PERSON 1",
"email": "example@test.com"
},
"campaign_name": "Campaign 1",
"enrollment_date": "2023-02-15T15:10:57.000Z",
"start_date": null,
"completion_date": null,
"status": "Not Started",
"time_spent": 0,
"policy_acknowledged": false
},
{
"enrollment_id": 1223525,
"content_type": "Store Purchase",
"module_name": "Module 1",
"user": {
"id": 111112,
"first_name": "TEST",
"last_name": "PERSON 2",
"email": "example@test.com"
},
"campaign_name": "Campaign 1",
"enrollment_date": "2023-02-15T15:10:57.000Z",
"start_date": "2023-02-18T19:43:31.000Z",
"completion_date": "2023-02-18T20:08:04.000Z",
"status": "Passed",
"time_spent": 1574,
"policy_acknowledged": false
}
]
Here are my original Query Parameters:
But the problem is I canโt figure out how to increment the page parameter in the pagination section. Iโve figured out how to trigger pagination via the $entity.length comparison, but not how to grab the next page:
Any help is greatly appreciated.
โ02-23-2023 10:22 AM
How do you know there are more pages to request? Iโm only seeing two results in the first page, but you specified 500 results per page. Doesnโt that mean there are no more results?
โ02-23-2023 10:41 AM
I truncated the results for simplicity to show thereโs no total_results or pages in my response. I know there are 3 pages of data (~1300 results) because Iโve run API request through postman.
To answer your first question, the only way I know thereโs more to request is if there are 500 results returned on a page (max defined by the API), then Iโd try to pull the next page. Rinse and repeat until the results returned is not 500. I realize thereโs a potential for a campaign to have a number of results divisible by 500 and will make a call that will result in 0 responses.