This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
FAQ
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Best way to process millions of records in REST GET API call without Read time out error

amardeep2021
New Contributor III

‎12-08-2021 09:19 AM

Hi,

I am trying to process millions of records by using REST GET snap API call. I am getting Read time out error. Is there a best way to bring in millions of records in REST call without facing Read time out failure during execution? Currently I am using Mapper–>REST GET–>Mapper for further processing.

Please let me know with your tips.

Regards,
Amar

Reply
10 REPLIES 10

bojanvelevski
Valued Contributor

‎12-08-2021 10:00 AM

Does increasing the Read Timeout works for you?

0 Kudos
Reply

amardeep2021
New Contributor III

‎12-08-2021 10:14 AM

No. It failed after 6 hours of execution. I am wondering what could be the best solution to overcome this Read time out failure.

0 Kudos
Reply

bojanvelevski
Valued Contributor
In response to amardeep2021

‎12-08-2021 10:30 AM

It’s highly likely that the API supports pagination, or some other way of chunking the requested data. That’s where I would go first. It will be helpful if you share the name of the API you are using, that’s it if you are able to.

0 Kudos
Reply

amardeep2021
New Contributor III

‎12-08-2021 10:58 AM

This is the API account setting I am using. Basic Auth Account.

0 Kudos
Reply