Dear All,
I am currently using the REST GET snap in Snaplogic to call a REST OData API. There are 1k tables and it is giving me next link when calling from postman.
However, I am facing an issue where I can only retrieve 10,000 rows despite there being over 500,000 records in the tables. I would greatly appreciate it if someone could help me understand why this limitation is occurring.
Is it possible to get all the data at one go ?
Thanks in advance.
DP
Deepanshu,
1. Are you trying to retrieve the rows in a table (which is 500K) or all the tables (1K)?
2. The logic isn't much different with what you will do in postman, you will need to use pagination and follow the next link. Is your snap configured for pagination?
3. Also the recommendation from snaplogic is to start using http snap. Please switch if possible.
I am having 1k tables and inside the tables we have records in lacs but it is only showing 10k for all the records and i have replaced my REST GET snap with http client but is there any way I can use nextlink for all the tables in one snap itself ?
deepanshu_1 , can you provide more info? show an sample response from one of the calls by removing all sensitive data?
Hi deepanshu_1 ,
Good day, I don't know if this setting affects the REST/HTTP client snaps
Also, with that much record have you tried receiving the response as binary and save it locally in pipeline temp folder then parse the file that you receive
Thanks,
EmEm
You want to use the HTTP client snap for this. It will handle pagination easier than the REST snap pack and should be used instead of the REST Snap for all new development. Ref: https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2614591489/HTTP+Client
Also, you need to confirm the source does not have any API limits. Some endpoints throttle traffic and limit the number of calls per minute or rows of data per 24 hours. Make sure the source is not ignoring your requests.
