Thanks Taylor and Viktor.
Viktor,
I am making 1 REST GET API call to bring in million records to process them. Can you please elaborate little more on group by? I am currently using REST GET with url and trying to process required fields using mapper after that. You are suggesting to use a group by snap after REST GET and connect that to execute pipeline?
Please let me know.
Regards,
Amardeep Shanmugam.
My suggestion above was only if you are making large number of requests to the API, since you are making one request that would not work.
In situation like yours, it can only be done something with the things that API is providing.
You can check if the API have some parameter where you will specify wanted columns, or
see if the API have pagination.
It is most likely to have pagination because as you said you are bringing million of records and that is large number of data to be passed in one request.
If accepts pagination, check this blog, explains how to make pagination with REST Get snap.
