cancel
Showing results for 
Search instead for 
Did you mean: 

Running Rest API in a loop where Page count is fixed variable

adityamohanty
New Contributor II

Step -1 

Rest API to get the collection info looks like 

https://stage-data.xyz.com/v1/xyz-api/collections/models/info?page_size=100

Output:

{
    "total_count"5099,
    "page_count"51,
    "page_size"100
}
Step -2
Rest API to get the model looks like
 
Output:
It will give model information of 1st page with 100 records.
 
Question - I want to run it 51 times (based on page count) in a loop to load entire records 
 
Solution:
 
I captured the page count & total count through mapper and my Get Rest Snap looks like 
HTTP Header : As per Need
Has next : snap.in.totalCount == $page_count
Next URL: "https://stage-data.xyz.com/v1/xyz-api/collections/models/" + (snap.in.totalCount + 1) + "?page_size=100" + '?skip=' + (snap.in.totalCount + 1).toString()
 
I am able to load only 100 records to my target. Please advise
1 ACCEPTED SOLUTION

Hello again,

As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

Please refer to the attached pipeline and let me know if this helps you.

Regards,

Aleksandar.

Please let me know if this helps you!

Best Regards,
Aleksandar.

View solution in original post

5 REPLIES 5

AleksandarAngel
Contributor III

Hello @adityamohanty,

You could with the following approach:

  1. Generate a document per page. (Dynamically based on the page count and page size from the initial response)

 

 

sl.range(1,$page_count+1,1).map(x => {"page":x, "page_size":$page_size})

 

 

  • Split the array using JSON Splitter Snap.
  • REST GET request per each document.

I chose this approach since I'm not aware of the format of the response of the models endpoint. If there are pagination information, you could achieve the same using REST GET Snap or HTTP Client Snaps with proper pagination enabled.

You can refer to the pipeline attached below and let me know if this helps you.

Regards,

Aleksandar.

Please let me know if this helps you!

Best Regards,
Aleksandar.

adityamohanty
New Contributor II

Hello @AleksandarAngel 

Thank you so much for the reply. I understood your approach but not sure how does it help to run it in a loop because following "Has Next" and "next url" is blank in REST GET request in your pipeline (Screenshot attached)

And I have few parameters and token which has been used to call the API and I can successfully call it out through mapper and used in REST GET snap. But when passing JSON Splitter as per the suggestion between Mapper and REST GET to  split the array of pages, it's not capturing my parameters and token value to the REST GET Input. The only output of JSON Splitter, I can see is Page & Page Size. Any advise ?

Hello again,

As I mentioned above, this is not how a typical pagination would look like using the REST GET / HTTP Client Snap, since I'm not aware about any information in the response that can be used to set the Has Next and Next URL properties. 

And since that you have the total number of pages in the info request, this approach will generate as many documents (requests) as you have pages.

If your parameters are static you can map them in the Map - Pages Mapper Snap and pass them in the JSON Splitter in the Include Paths section so you can use them in the REST GET Snap.

Please refer to the attached pipeline and let me know if this helps you.

Regards,

Aleksandar.

Please let me know if this helps you!

Best Regards,
Aleksandar.

Thank You @AleksandarAngel. It worked for me. I just wanted to check one more thing. When we create no of documents to hit the API that no of times. Does it run sequentially ?

I mean 1 document hits the API and once it gets the data then 2 document hits the API ?