Best way to process millions of records in REST GET API call without Read time out error
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎12-08-2021 09:19 AM
Hi,
I am trying to process millions of records by using REST GET snap API call. I am getting Read time out error. Is there a best way to bring in millions of records in REST call without facing Read time out failure during execution? Currently I am using Mapper–>REST GET–>Mapper for further processing.
Please let me know with your tips.
Regards,
Amar

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎12-08-2021 10:00 AM
Does increasing the Read Timeout works for you?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎12-08-2021 10:14 AM
No. It failed after 6 hours of execution. I am wondering what could be the best solution to overcome this Read time out failure.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎12-08-2021 10:30 AM
It’s highly likely that the API supports pagination, or some other way of chunking the requested data. That’s where I would go first. It will be helpful if you share the name of the API you are using, that’s it if you are able to.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎12-08-2021 10:58 AM
This is the API account setting I am using. Basic Auth Account.
