Forum Discussion

arvindnsn's avatar
arvindnsn
Contributor
7 years ago

How to loop thru a REST snap

Hello,

I am trying to load transaction data from a REST API end point. The API has pagination as required Input parameter. I am using the REST GET to fetch the data and the Https URL should be as below.

https://xxxxxx–xxx/transactions?datestart=2018-11-19T00:00:00&dateend=2018-11-20T00:00:00&pageSize=1000&pageNo=1.

I am trying to design a process which loops thru till the REST API call returns null data set.

For example, if the above REST URL gives output, I have to call the same URL again with the pageNo input parameter as 2 and if it provides a data set, then 3 and on and on. The loop has to end when the REST API comes out with a null data.

Any help to solve this would be highly appreciated and also would like to know if looping is the only option and right approach.

Thanks
Aravind N

26 Replies

  • dwhite's avatar
    dwhite
    Employee
    8 years ago

    Replace your two scripts with mappers should net you some performance gain. You don’t need to use a script to make a string of json.

    Script in “Split events” pipeline which is doing

    data = self.input.next()
    newData = {}
    newData[‘record’] = data
    newData[‘sequence_id’] = data[‘sequence’][‘id’]
    newData[‘event_id’] = data[‘event_id’]
    self.output.write(newData)

    replace with a mapper like

    Script in write events is like
    data = self.input.next()
    self.output.write(data[‘record’])

    replace with a mapper like

    • aditya_sharma's avatar
      aditya_sharma
      New Contributor III
      8 years ago

      Thanks Dwhite

      I’ll make that change, but I checked the average execution time for both the scripts , it’s very less (0.141 sec).
      Also, how should we decide what to set in pool size property of Pipeline Execute snap.

      Thanks