Pagination in REST API
A very good explanation of how to implement pagination in REST API in this below blog post: SnapLogic – 4 Feb 16 REST GET and the SnapLogic Public APIs for Pipeline Executions As a part of a wider analytics project I’m working on, analyzing runtime information from the SnapLogic platform, I chose ... When I try to implement the same(I am extracting data from survey monkey api), I have noticed that when the “Next URL” is executing it is ignoring the query parameters provided.As shown in below image, when service URL is executing it’s getting right records considering all input query parameters i.e total:3043 but when “Next URL” is hitting it is extracting all records i.e. total : 61896 How to include query parameters in “Next URL” to get required data?
Smartsheet rest api pagination
Hi , We are integrating smartsheet rest api’s in snaplogic. We get only totalrecordscount in the response. there is no other things comes up in response. I need to apply pagination . In paramters i can use page and pageSize . Page is something which page we want to access and pageSize is 100 by default. These 2 paramters are not part of the response. I am not sure how I can create hasNext and nextUrl to paginate and fetch all 77k records. I am attaching the screenshots of rest snap settings .I am unable to loop. it is like never ending. please suggest. https://smartsheet.redoc.ly/tag/reports#operation/getReport offset is a parameter i gave as 0 for incrementing in nexUrl page
Github API Pagination
Hey community, This is my first attempt to get pagination working on a rest call but I’m a bit stuck on how to deal with the ‘Has Next’ and ‘Next URL’ options. The API response contains the next URL but it also contains the following URL. What do I need to put in each of the pagination options to get this to work?
REST pagination works in validate but fails in executions
Hi, I’m having an issue running a pipeline with a REST GET with a next URL. When I run a validation, the SNAP works fine, and I even see multiple responses indicating the next URL seems to be returned. When I run an execution however, the SNAP aborts and I get the following error: “com.snaplogic.snap.api.SnapDataException: REST API service endpoint returned error result: status code = 200 at com.snaplogic.snap.api.rest.RestResponseProcessor.process(RestResponseProcessor.java:336) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:369) at com.snaplogic.snap.api.rest.RestRequestExecutor.executeOnce(RestRequestExecutor.java:303) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:186) at com.snaplogic.snap.api.rest.RestRequestExecutor.execute(RestRequestExecutor.java:169) at com.snaplogic.snaps.rest.Get.process(Get.java:147) at com.snaplogic.snap.api.ExecutionUtil.process(ExecutionUtil.java:106) at com.snaplogic.snap.api.ExecutionUtil.executeForDocument(ExecutionUtil.java:118) at com.snaplogic.snap.api.ExecutionUtil.execute(ExecutionUtil.java:81) at com.snaplogic.snap.api.SimpleSnap.execute(SimpleSnap.java:70) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:803) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:585) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:869) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:435) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:117) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.snaplogic.cc.snap.common.SnapStreamException: Exception while writing document to the stream at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.write(DocumentOutputViewImpl.java:251) at com.snaplogic.snap.api.OutBoundViewsImpl.write(OutBoundViewsImpl.java:221) at com.snaplogic.snap.api.OutBoundViewsImpl.write(OutBoundViewsImpl.java:257) at com.snaplogic.snap.api.rest.RestResponseProcessor.writeToOutput(RestResponseProcessor.java:630) at com.snaplogic.snap.api.rest.RestResponseProcessor.processEntity(RestResponseProcessor.java:659) at com.snaplogic.snap.api.rest.RestResponseProcessor.process(RestResponseProcessor.java:316) … 20 more Caused by: com.snaplogic.cc.jstream.AbortException: Attempt to write to closed view at com.snaplogic.cc.jstream.view.publisher.NullPublisher.publish(NullPublisher.java:53) at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.lambda$write$0(DocumentOutputViewImpl.java:224) at com.snaplogic.cc.snap.common.ViewImpl.timeCall(ViewImpl.java:658) at com.snaplogic.cc.snap.view.document.DocumentOutputViewImpl.write(DocumentOutputViewImpl.java:156) … 25 more Reason: Error occurred while parsing JSON :Attempt to write to closed view Resolution: Please check the headers of the request and make sure that json is returned by the endpoint.” I don’t really understand what is happening, but I’m guessing the output view is closing before the snap has finished? Is there any advice on how I can resolve this? Thank you.Connecting to Marketo with the REST Snap Pack
While SnapLogic’s REST OAuth account supports only OAuth 2.0, it does not work with Marketo’s OAuth implementation. To work with Marketo, you must authenticate manually using the REST Get Snap. In this pipeline, we pass the credentials in as pipeline parameters. Note: This method does expose your credentials in the pipeline. Authorization To simplify the process, define the following pipeline parameters url: the REST API URL for your Marketo instance, like: https://xxx-xxx-xxx.mktorest.com clientID: The clientID for API access. clientKey: The client secret for API access. Add a REST Get Snap (labeled Marketo Login here) and configure as follows: For Service URL, toggle on the Expression button ( = ) and set the field to: _url + '/identity/oauth/token?grant_type=client_credentials&client_id=' + _clientID + '&client_secret=' +_clientKey Remove the input view. Validate the Snap and it will a return a response that contains an access_token and scope. In this example, we follow the REST Get with a Mapper Snap to map the token outside of the array. Using the Access Token In subsequent Snaps, we pass this token as a header, rather than a query parameter because it simplifies paged operations such as Get Lead Changes. Here’s an example of a simple call which does this. For Service URL, toggle on the Expression button ( = ) and set the field to: _url + '/rest/v1/activities/types.json' Under HTTP Header, set Key to Authorization and Value with the Expression button ( = ) toggled on to ‘Bearer ‘ + $accessToken Paged Operations When you get to more complex operations, such as getting lead changes, you need to make two API calls: the first creates a paging token, and the second uses the paging token typically with the paging mechanism enabled in our REST GET Snap. Get Paging Token In this REST Get Snap (renamed Get Paging Token for clarity) is where you specify the query parameters. For instance, if you want to get lead changes since a particular date, you’d pass that in via “sinceDateTime”. The example provided uses a literal string, but could be a pipeline parameter or ideally one of a Date objects formatted to match what Marketo expects. _url + '/rest/v1/activities/pagingtoken.json' Configure Paging Mechanism When calling Get Leads (via a REST GET Snap), a few things to bear in mind: You need to pass “nextPageToken” as a query parameter, along with the fields you want back. Ideally, the list of fields should be in a pipeline parameter because they appear twice in this configuration. The leads will be returned in $entity.result, which is an array. This field will not exist if there are no results, so you need to enable “Null safe” on a Splitter Snap after this REST Get. Paging expressions for the REST Get Snap are: Has next: $entity.moreResult == true Next URL: '%s/rest/v1/activities/leadchanges.json?nextPageToken=%s&fields=firstName,lastName'.sprintf( _url, $entity.nextPageToken ) API Throttling Marketo throttles API calls. Their documentation says “100 API calls in a 20 second window”. Since our REST Snap paging now includes an option to wait for X seconds or milliseconds between requests, use it whenever you are retrieving paginated results. Downloads Marketo REST.slp (14.7 KB)
Rest GET - Next URL with page=# syntax with no metadata
Hello, I’m calling a REST API which requires me to increment a ‘page’ value in order to get all the results. The issue is the API doesn’t return any metadata. For example: 1st call is: https:domain.com/object/page/1 Results: { entity:[ {…},{…},… ] } Has next: typeof $entity == ‘array’ && $entity.length > 0 Next URL: https:domain.com/object/page/2 The problem is how to I compute the 2 (then 3,4,5… etc) Is it possible to increment a value by 1 on every subsequent call? Or is it possible to access a variable that counts how many times the call has been made? Thanks!
Rest Get Pagination Issue
Hello, I am having issues with the Rest GET Pagination mechanism. The Pagination works fine in the validate mode, but doesn’t traverse after 1st page when i actually run the pipeline. This is so weird that i am able to see the pagination working fine in the “Validate&Execute” mode, but failing to move after the 1st page when i actually run the Pipeline. Below is the screenshot of the logic i use for pagination. Postman_output is the screenshot from Postman to show the API structure. Snap_output is the screenshot from SnapLogic Pipeline for Pagination. I have tried using the pagination intervals from 1 to 25 but in vain. I would really appreciate if any reason or solution is available. Thanks Aravind N
How to Access data from an REST API which has pagination as input parameters
Hello, I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below. https://xxxxxx--xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=3&pageNo=1. The issue is, the API doesn’t really give the total count of data and hence not able to breakdown the pagesize and page number. Question 1: How can i fetch data with the above scenario if there are huge volume of unknown number of records? Question 2: If i get to know the total count, Do i have to run the API for 2 passes with 1st pass to get total count and second pass to have a logic to loop thru the pagination dynamically? If so, What would be the best practice? Thanks Aravind N
