Running Rest API in a loop where Page count is fixed variable

[adityamohanty]

Step -1 

Rest API to get the collection info looks like 

https://stage-data.xyz.com/v1/xyz-api/collections/models/info?page_size=100

Output:

{

    "total_count": 5099,

    "page_count": 51,

    "page_size": 100

}

Step -2

Rest API to get the model looks like

https://stage-data.xyz.com/v1/xyz-api/collections/models/1?page_size=100

Output:

It will give model information of 1st page with 100 records.

Question - I want to run it 51 times (based on page count) in a loop to load entire records 

Solution:

I captured the page count & total count through mapper and my Get Rest Snap looks like 

Service URL :  https://stage-data.xyz.com/v1/xyz-api/collections/models/1?page_size=100

HTTP Header : As per Need

Has next : snap.in.totalCount == $page_count

Next URL: "https://stage-data.xyz.com/v1/xyz-api/collections/models/" + (snap.in.totalCount + 1) + "?page_size=100" + '?skip=' + (snap.in.totalCount + 1).toString()

I am able to load only 100 records to my target. Please advise

[AleksandarAngel]

Hello @adityamohanty,

You could with the following approach:

1. Generate a document per page. (Dynamically based on the page count and page size from the initial response)

sl.range(1,$page_count+1,1).map(x => {"page":x, "page_size":$page_size})

• Split the array using JSON Splitter Snap.
• REST GET request per each document.

I chose this approach since I'm not aware of the format of the response of the models endpoint. If there are pagination information, you could achieve the same using REST GET Snap or HTTP Client Snaps with proper pagination enabled.

You can refer to the pipeline attached below and let me know if this helps you.

Regards,

Aleksandar.

[adityamohanty]

Hello @AleksandarAngel 

Thank you so much for the reply. I understood your approach but not sure how does it help to run it in a loop because following "Has Next" and "next url" is blank in REST GET request in your pipeline (Screenshot attached)

And I have few parameters and token which has been used to call the API and I can successfully call it out through mapper and used in REST GET snap. But when passing JSON Splitter as per the suggestion between Mapper and REST GET to  split the array of pages, it's not capturing my parameters and token value to the REST GET Input. The only output of JSON Splitter, I can see is Page & Page Size. Any advise ?

[AleksandarAngel · Accepted solution]

Hello again,

As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

Please refer to the attached pipeline and let me know if this helps you.

Regards,

Aleksandar.

[adityamohanty]

Thank You @AleksandarAngel. It worked for me. I just wanted to check one more thing. When we create no of documents to hit the API that no of times. Does it run sequentially ?

I mean 1 document hits the API and once it gets the data then 2 document hits the API ?