Pagination in REST API
A very good explanation of how to implement pagination in REST API in this below blog post: SnapLogic – 4 Feb 16 REST GET and the SnapLogic Public APIs for Pipeline Executions As a part of a wider analytics project I’m working on, analyzing runtime information from the SnapLogic platform, I chose ... When I try to implement the same(I am extracting data from survey monkey api), I have noticed that when the “Next URL” is executing it is ignoring the query parameters provided.As shown in below image, when service URL is executing it’s getting right records considering all input query parameters i.e total:3043 but when “Next URL” is hitting it is extracting all records i.e. total : 61896 How to include query parameters in “Next URL” to get required data?
Smartsheet rest api pagination
Hi , We are integrating smartsheet rest api’s in snaplogic. We get only totalrecordscount in the response. there is no other things comes up in response. I need to apply pagination . In paramters i can use page and pageSize . Page is something which page we want to access and pageSize is 100 by default. These 2 paramters are not part of the response. I am not sure how I can create hasNext and nextUrl to paginate and fetch all 77k records. I am attaching the screenshots of rest snap settings .I am unable to loop. it is like never ending. please suggest. https://smartsheet.redoc.ly/tag/reports#operation/getReport offset is a parameter i gave as 0 for incrementing in nexUrl page
Github API Pagination
Hey community, This is my first attempt to get pagination working on a rest call but I’m a bit stuck on how to deal with the ‘Has Next’ and ‘Next URL’ options. The API response contains the next URL but it also contains the following URL. What do I need to put in each of the pagination options to get this to work?
REST pagination works in validate but fails in executions
Hi, I’m having an issue running a pipeline with a REST GET with a next URL. When I run a validation, the SNAP works fine, and I even see multiple responses indicating the next URL seems to be returned. When I run an execution however, the SNAP aborts and I get the following error: “com.snaplogic.snap.api.SnapDataException: REST API service endpoint returned error result: status code = 200 at com.snaplogic.snap.api.rest.RestResponseProcessor.process(RestResponseProcessor.java:336) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:369) at com.snaplogic.snap.api.rest.RestRequestExecutor.executeOnce(RestRequestExecutor.java:303) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:186) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:169) at com.snaplogic.snaps.rest.Get.process(Get.java:147) at com.snaplogic.snap.api.ExecutionUtil.process(ExecutionUtil.java:106) at com.snaplogic.snap.api.ExecutionUtil.executeForDocument(ExecutionUtil.java:118) at com.snaplogic.snap.api.ExecutionUtil.execute(ExecutionUtil.java:81) at com.snaplogic.snap.api.SimpleSnap.execute(SimpleSnap.java:70) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:803) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:585) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:869) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:435) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:117) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.snaplogic.cc.snap.common.SnapStreamException: Exception while writing document to the stream at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.write(DocumentOutputViewImpl.java:251) at com.snaplogic.snap.api.OutBoundViewsImpl.write(OutBoundViewsImpl.java:221) at com.snaplogic.snap.api.OutBoundViewsImpl.write(OutBoundViewsImpl.java:257) at com.snaplogic.snap.api.rest.RestResponseProcessor.writeToOutput(RestResponseProcessor.java:630) at com.snaplogic.snap.api.rest.RestResponseProcessor.processEntity(RestResponseProcessor.java:659) at com.snaplogic.snap.api.rest.RestResponseProcessor.process(RestResponseProcessor.java:316) … 20 more Caused by: com.snaplogic.cc.jstream.AbortException: Attempt to write to closed view at com.snaplogic.cc.jstream.view.publisher.NullPublisher.publish(NullPublisher.java:53) at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.lambda$write$0(DocumentOutputViewImpl.java:224) at com.snaplogic.cc.snap.common.ViewImpl.timeCall(ViewImpl.java:658) at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.write(DocumentOutputViewImpl.java:156) … 25 more Reason: Error occurred while parsing JSON :Attempt to write to closed view Resolution: Please check the headers of the request and make sure that json is returned by the endpoint.” I don’t really understand what is happening, but I’m guessing the output view is closing before the snap has finished? Is there any advice on how I can resolve this? Thank you.
Rest GET - Next URL with page=# syntax with no metadata
Hello, I’m calling a REST API which requires me to increment a ‘page’ value in order to get all the results. The issue is the API doesn’t return any metadata. For example: 1st call is: https:domain.com/object/page/1 Results: { entity:[ {…},{…},… ] } Has next: typeof $entity == ‘array’ && $entity.length > 0 Next URL: https:domain.com/object/page/2 The problem is how to I compute the 2 (then 3,4,5… etc) Is it possible to increment a value by 1 on every subsequent call? Or is it possible to access a variable that counts how many times the call has been made? Thanks!
Rest Get Pagination Issue
Hello, I am having issues with the Rest GET Pagination mechanism. The Pagination works fine in the validate mode, but doesn’t traverse after 1st page when i actually run the pipeline. This is so weird that i am able to see the pagination working fine in the “Validate&Execute” mode, but failing to move after the 1st page when i actually run the Pipeline. Below is the screenshot of the logic i use for pagination. Postman_output is the screenshot from Postman to show the API structure. Snap_output is the screenshot from SnapLogic Pipeline for Pagination. I have tried using the pagination intervals from 1 to 25 but in vain. I would really appreciate if any reason or solution is available. Thanks Aravind N
How to Access data from an REST API which has pagination as input parameters
Hello, I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below. https://xxxxxx--xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=3&pageNo=1. The issue is, the API doesn’t really give the total count of data and hence not able to breakdown the pagesize and page number. Question 1: How can i fetch data with the above scenario if there are huge volume of unknown number of records? Question 2: If i get to know the total count, Do i have to run the API for 2 passes with 1st pass to get total count and second pass to have a logic to loop thru the pagination dynamically? If so, What would be the best practice? Thanks Aravind N
