Hi,
I am trying to process millions of records by using REST GET snap API call. I am getting Read time out error. Is there a best way to bring in millions of records in REST call without facing Read time out failure during execution? Currently I am using Mapper–>REST GET–>Mapper for further processing.
Please let me know with your tips.
Regards,
Amar
Hi @amardeep2021 ,
As @bojanvelevski said, pagination should do the job.
Check the documentation about the API you are using and see if it supports pagination.
Here REST Get you can find more about it.
Look for the attributes “Has Next” and “Next URL”.
Regards
Does increasing the Read Timeout works for you?
No. It failed after 6 hours of execution. I am wondering what could be the best solution to overcome this Read time out failure.
It’s highly likely that the API supports pagination, or some other way of chunking the requested data. That’s where I would go first. It will be helpful if you share the name of the API you are using, that’s it if you are able to.
This is the API account setting I am using. Basic Auth Account.
Hi All,
Thanks for your suggestions.
Yes REST GET has ‘Has Next’ and ‘Next URL’
Regards,
Amardeep Shanmugam.
I am trying to pull data from Workday API. I don’t have ‘Has Next’ and ‘Next URL’ expression, it’s blank. Just to clarify. Does this gives anyone on what should be the solution.? Please let me know.
Moreover, I have only these information in output preview of REST GET snap during validation. I don’t see offset in there. Please let me know with this information how can I implement pagination to avoid failure?
Hi @amardeep2021,
By default the ‘Has Next’ and ‘Next URL’ are blank because you need to specify the ‘Has Next’ condition and ‘Next URL’ to be called. And as I can see, this is an empty response you are getting. Which endpoint are you trying to call ?
And, you should really look into the documentation of the API itself. That’s the only way to find how you can use pagination.
@amardeep2021, it looks like you’re calling a Workday RaaS (web-enabled) report. These Workday reports do not natively support pagination. The Workday Community site has multiple Brain Storm requests for a pagination feature for the reports, but Workday has not addressed these requests.
With that said, you may can work with your Workday Report team/developers to creatively develop a report with pagination/chunking capabilities.
For example, if the report data source records have timestamps or effective dating, the report team could provide prompt fields whereby you can include start and end times in the URL query string which would allow you to chunk the data by time ranges. Depending on the data you’re pulling in, there may be other creative ways to chunk the data to provide pagination capabilities.
Likewise, if this post is related to your other post about reducing the columns pulled in from the API using REST GET, the report team can reduce the number of columns in the report output.
