Forum Discussion

adityamohanty's avatar
adityamohanty
New Contributor II
2 years ago
Solved

Running Rest API in a loop where Page count is fixed variable

Step -1 

Rest API to get the collection info looks like 

https://stage-data.xyz.com/v1/xyz-api/collections/models/info?page_size=100

Output:

{
    "total_count"5099,
    "page_count"51,
    "page_size"100
}
Step -2
Rest API to get the model looks like
 
Output:
It will give model information of 1st page with 100 records.
 
Question - I want to run it 51 times (based on page count) in a loop to load entire records 
 
Solution:
 
I captured the page count & total count through mapper and my Get Rest Snap looks like 
HTTP Header : As per Need
Has next : snap.in.totalCount == $page_count
Next URL: "https://stage-data.xyz.com/v1/xyz-api/collections/models/" + (snap.in.totalCount + 1) + "?page_size=100" + '?skip=' + (snap.in.totalCount + 1).toString()
 
I am able to load only 100 records to my target. Please advise
  • Aleksandar_A's avatar
    Aleksandar_A
    2 years ago

    Hello again,

    As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

    And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

    If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

    Please refer to the attached pipeline and let me know if this helps you.

    Regards,

    Aleksandar.

5 Replies

  • I am not sure what you mean by fetching the log for the pipeline. Do you mean runtime statistics for the pipeline runtimes? Do you mean debug logs generated on the Snaplex nodes? Runtime statistics are available through runtime APIs. Debug logs are available using the logs APIs.

    • wcl3y2's avatar
      wcl3y2
      New Contributor III

      Hello, I’m talking about the log API located here: https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438932/Log+API.

      When passing in the RUUID for the subscriber event, I don’t get any results. I’m really just looking for any way to get verbose logging for the Salesforce subscriber snap. We have instances where it loses connection with no errors that we can see, and we’re trying to troubleshoot.