Forum Discussion

adityamohanty's avatar
adityamohanty
New Contributor II
2 years ago
Solved

Running Rest API in a loop where Page count is fixed variable

Step -1 

Rest API to get the collection info looks like 

https://stage-data.xyz.com/v1/xyz-api/collections/models/info?page_size=100

Output:

{
    "total_count": 5099,
    "page_count": 51,
    "page_size": 100
}
Step -2
Rest API to get the model looks like
 
Output:
It will give model information of 1st page with 100 records.
 
Question - I want to run it 51 times (based on page count) in a loop to load entire records 
 
Solution:
 
I captured the page count & total count through mapper and my Get Rest Snap looks like 
HTTP Header : As per Need
Has next : snap.in.totalCount == $page_count
Next URL: "https://stage-data.xyz.com/v1/xyz-api/collections/models/" + (snap.in.totalCount + 1) + "?page_size=100" + '?skip=' + (snap.in.totalCount + 1).toString()
 
I am able to load only 100 records to my target. Please advise
  • Aleksandar_A's avatar
    Aleksandar_A
    2 years ago

    Hello again,

    As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

    And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

    If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

    Please refer to the attached pipeline and let me know if this helps you.

    Regards,

    Aleksandar.

5 Replies

  • Aleksandar_A's avatar
    Aleksandar_A
    Contributor III
    2 years ago

    Hello adityamohanty,

    You could with the following approach:

    1. Generate a document per page. (Dynamically based on the page count and page size from the initial response)

     

     

    sl.range(1,$page_count+1,1).map(x => {"page":x, "page_size":$page_size})

     

     

    • Split the array using JSON Splitter Snap.
    • REST GET request per each document.

    I chose this approach since I'm not aware of the format of the response of the models endpoint. If there are pagination information, you could achieve the same using REST GET Snap or HTTP Client Snaps with proper pagination enabled.

    You can refer to the pipeline attached below and let me know if this helps you.

    Regards,

    Aleksandar.

  • adityamohanty's avatar
    adityamohanty
    New Contributor II
    2 years ago

    Hello Aleksandar_A 

    Thank you so much for the reply. I understood your approach but not sure how does it help to run it in a loop because following "Has Next" and "next url" is blank in REST GET request in your pipeline (Screenshot attached)

    And I have few parameters and token which has been used to call the API and I can successfully call it out through mapper and used in REST GET snap. But when passing JSON Splitter as per the suggestion between Mapper and REST GET to  split the array of pages, it's not capturing my parameters and token value to the REST GET Input. The only output of JSON Splitter, I can see is Page & Page Size. Any advise ?

    • Aleksandar_A's avatar
      Aleksandar_A
      Contributor III
      2 years ago

      Hello again,

      As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

      And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

      If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

      Please refer to the attached pipeline and let me know if this helps you.

      Regards,

      Aleksandar.

      • adityamohanty's avatar
        adityamohanty
        New Contributor II
        2 years ago

        Thank You Aleksandar_A. It worked for me. I just wanted to check one more thing. When we create no of documents to hit the API that no of times. Does it run sequentially ?

        I mean 1 document hits the API and once it gets the data then 2 document hits the API ?