1 ACCEPTED SOLUTION
@koryknick, Thank you for your help. I was provided a solution by a coworker.
It's essentially that pipeline solution you provided HTTP Client Pagination with a few tweaks. I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap "after" : $splitValue, "limit" : 100 Then I was getting subsequent data. But the data was duplicating. So I modified the map for pagination as follows:
sl.range(1, parseInt($entity.total), 100) -I also changed the 1st http snap to just give me 1 record just so that I get the total and I process from page 1 to the end in the rest of the pipeline so I don't do a union.
And it worked! I am now getting subsequent data and it the records are not duplicating.
