Best way to process millions of records in REST GET API call without Read time out error

[amardeep2021]

Hi,

I am trying to process millions of records by using REST GET snap API call. I am getting Read time out error. Is there a best way to bring in millions of records in REST call without facing Read time out failure during execution? Currently I am using Mapper–>REST GET–>Mapper for further processing.

Please let me know with your tips.

Regards,
Amar

[marjan_karafilo]

Hi @amardeep2021 ,

As @bojanvelevski said, pagination should do the job.
Check the documentation about the API you are using and see if it supports pagination.

Here REST Get you can find more about it.
Look for the attributes “Has Next” and “Next URL”.

Regards

[amardeep2021]

Hi All,

Thanks for your suggestions.

@marjan.karafiloski ,

Yes REST GET has ‘Has Next’ and ‘Next URL’

Regards,
Amardeep Shanmugam.

[amardeep2021]

I am trying to pull data from Workday API. I don’t have ‘Has Next’ and ‘Next URL’ expression, it’s blank. Just to clarify. Does this gives anyone on what should be the solution.? Please let me know.

[amardeep2021]

Moreover, I have only these information in output preview of REST GET snap during validation. I don’t see offset in there. Please let me know with this information how can I implement pagination to avoid failure?

[j_angelevski]

Hi @amardeep2021,

By default the ‘Has Next’ and ‘Next URL’ are blank because you need to specify the ‘Has Next’ condition and ‘Next URL’ to be called. And as I can see, this is an empty response you are getting. Which endpoint are you trying to call ?

And, you should really look into the documentation of the API itself. That’s the only way to find how you can use pagination.