Need to chunk files and hit the api for each record

Question

Need to chunk files and hit the api for each record. Restriction on api to hit only 1000 records per request. The input has millions of records. how to handle this in same pipeline

abhishek_soni37 · Answer

Hi @reachsenthil
Assuming you are using Rest Post to send data to the API, you can define the batch size.
Have a look at Rest POST.

reachsenthil · Answer

Hi @Soni37 , I am using REST GET to hit the url with parameters and get the response and load to table. so the restriction is to hit api for 1000 records per request

abhishek_soni37 · Answer

reachsenthil:

REST GET

You can use Group By N snap to create a batch of requests before feeding it to REST Get snap.
Test_Expression_2023_06_22.slp (3.8 KB)
The response will look like this: In my case, I created a group of 8 records.

Let me know if it works. 🙂

Forum Discussion

Need to chunk files and hit the api for each record

3 Replies

Recent Discussions

Pagination and nextCursor in header

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?