cancel
Showing results for 
Search instead for 
Did you mean: 

HTTP client

rjapala
New Contributor III

**Task :**We have n number of records in the below URL, we have to pull all the records. For each request there should be limit of 10 records for testing.

How to use Pagination for this Task,

Note: In my client URL there is no offset, start or limit.

For Example:
First request 0-10
Second request 11-20

It should take dynamically the next 10 records for each request and send for testing.

Please help me on this task and how to use pagination concept.

Thanks in advance !!

@ddellsperger @dmiller

8 REPLIES 8

dmiller
Former Employee

I haven’t used HTTP Client much yet, so I’m not sure, but I know there a couple other posts on pagination in this Snap:


Diane Miller

ddellsperger
Moderator
Moderator

With HTTP Client, we provide maybe a different way to do this, but it’s still possible though likely a bit more manual. The Http Client has a pagination section (which you can expand - see the screenshot below)
The best way to explain these options is that the “Has next” input should evaluate to a boolean to determine if there are more pages to get or not (so in your case, if $entities.length < 10 would likely be what you want to enter here, that would break the pagination if you have fewer than 10 entries in your output document. The “Total pages to fetch” would be a secondary block for stopping pagination, so if you wanted to get all pages, you’d never set the “Total pages to fetch” if you want to limit the number of pages to however many are available up to 10, you’d set that value to 10. Override URI is the uri you’d have to request to get the next page, in this case (for offset paging) you’d have to do some work to get this working, if your original request url is something like https://example.com/api/output with query parameters of count 10, then you could either change to “reuse request parameters” and define the new ones there, or change the full url completely to something like "https://example.com/api/output?count=10&offset=" + (snap.out.totalCount * 10). This would allow your offset to increase by 10 for every output document produced, etc. You could do this either in the “Override URI” or in the “Override parameters” to support this capability. You’ll somewhat note that these are both only visible when the checkbox is checked, so that will determine which process it takes. Typically, “Override URI” is useful for cursor-based pagination and “Override parameters” are useful for count-based pagination assuming you externalize all of your query parameters and the triggers are able to be set via the query parameters rather than path parameters.
image

@ddellsperger  - 

How does the above work if there is no offset param? I'm following your post, but I'm not sure how to build the Next URI link when there is no offset option for me. I've been told the API I'm pulling from supports "cursor-style" pagination. No offset, limit, or start....

Any help on this would be greatly appreciated!

 

Chris

Cursor pagination is actually typically much more straightforward, you should typically have a "next url" or "next page cursor" and if there's no next page, that should be null or empty, in that case your has next would be something like $.entity.paging.nextCursor != null and you would likely use an override query parameter with something like cursor being equal to $.entity.paging.nextCursor. Typically, cursor pagination uses the cursor as a query parameter after the initial page, you'd have to see how the documentation claims to pass in the cursor, though.