Issue configuring pagination in HTTP Client snap

[jfpelletier]

Hello all,

I'm using a HTTP snap to get a list of tasks in a given project, and by default the server returns maximum 100 items per call. To get the items above 100, the endpoint has a query parameter "skip" that offsets the number of items to skip to reach to the desired page (the default is "0"), and another one labelled "top" which lets me specify the number of items to include inside the pages.

I have configured the snap as per the screenshot below, but that doesn't work, it's only pulling me in what seems to be an infinite loop (I'm getting a 429 - too many requests) after a couple of minutes... 😞 

I'm obviously missing something, but I can't figure it out... Is there anyone kind enough to help me out?

Thanks!

JF

[koryknick]

@jfpelletier - if you check the "Enable debug" in the snap, you will see a "_debug" field that contains some useful information to help diagnose what's going on.  In particular, look at the URL generated for each page call.  To temporarily limit the number of iterations, you can update your "Has next" to something like "snap.out.totalCount < 5 && $entity.itemCount > 100" which will stop the iterations at 5 so you can view the results.

My guess is that your "skip" value is the number to skip from zero, not the last iteration.  So that might need to be something like "100 * (snap.out.totalCount + 1)" - which would result in 100, 200, 300, ..., and so on until it reaches the end.

Hope this helps!

[jfpelletier]

Hello @koryknick ,

Thanks a lot for your reply! I have found the solution in your suggestion. I didn't know about the snap.out.totalCount counter, and I used it to count and increment the items passed to that I can control the flow with this "Has next" expression: $entity.itemCount - ((snap.out.totalCount + 1) * 100) > 0

It works like a charm, and I can even control the number of items per page as well.

Have a nice day, and thanks again for your help!

JF