Forum Discussion

ed_tc's avatar
ed_tc
New Contributor III
7 years ago
Solved

How to deal with paging in post response

Hello all, I have to deal with API with paging. In the body of the response I received not only data but also cursor(pointer to the next page of data). I used REST POST snap. The API was designed...
  • ed_tc's avatar
    ed_tc
    7 years ago


    Hello, not sure that I can, but here it is a skeleton.
    The main idea is if we received an error, we should recall the pipeline with the same parameters, if not we should call the pipeline with the same parameter except cursor( in my case). I got value for the cursor parameter and check it before next iteration. If it not null I recall pipeline with the new cursor + all parameters.
    Get next id’s list and recurse snap I used for call the pipeline again

brianrandolph's avatar
brianrandolph
New Contributor III
7 years ago

Rest GET has built in pagination feature. Looks like POST does not. Maybe you can build a Parent pipeline that repeatedly calls a child pipeline with the cursor?

We are working on a similar solution with Rest GET where a parent pipeline calls a child that gets one batch at a time. As long as there are more pages we call the child again.

  • ed_tc's avatar
    ed_tc
    New Contributor III
    7 years ago

    Thank you for responding. After some time I’ve implemented recursive pipeline which does almost the same as foreach loop in programming languages.

    • brianrandolph's avatar
      brianrandolph
      New Contributor III
      7 years ago

      any way you can share your solution? i’m working on similar pipeline, we are now getting empty results from a 500 or 403, so do you have any thoughts on that?

      • ed_tc's avatar
        ed_tc
        New Contributor III
        7 years ago


        Hello, not sure that I can, but here it is a skeleton.
        The main idea is if we received an error, we should recall the pipeline with the same parameters, if not we should call the pipeline with the same parameter except cursor( in my case). I got value for the cursor parameter and check it before next iteration. If it not null I recall pipeline with the new cursor + all parameters.
        Get next id’s list and recurse snap I used for call the pipeline again